The Testing Show: Making the “Full Stack” Tester

This week, we are joined by Jess Ingrassellino and Garry Heon to talk about where testing is going, and what the future of testing holds, going beyond Agile and dev-ops, or at least seeing where and how testers today will better be able to work and thrive in this brave new world. In the process of talking about that, we joked about the idea of a “full stack tester” to go with the increasing demand of “full stack developers”. Is there such a thing as a full stack tester? We weren’t sure, but if there was, we figured we could relate to the idea, and we’d be interested in seeing that idea develop, so to speak.

Panelists:

References:

• Meet the Company Whose Webcams Shut Down the Internet Last Friday
• Mirai (malware)
• Distributed Denial of Service
• Abbie Bangser
• A/B Testing
• Python Projects for Kids
• PyCon Education Summit 2017
• Agile Testing Days
• U.S. AgileTestingDays
• Agile and Beyond 2017

Transcript:

MICHAEL LARSEN: Welcome to the testing show, I’m Michael Larsen, your show producer and this week we have a panel of wild and wonderful guests. Let’s welcome our regulars, Justin Rohrman.

JUSTIN ROHRMAN: Good morning.

MICHAEL LARSEN: Our special guests, Jess Ingrasselino.

JESS INGRASSELLINO: Good morning.

MICHAEL LARSEN: Garry Heon

GARRY HEON: Good morning.

MICHAEL LARSEN: And of course our ever so active moderator, Mr. Matt Heusser, it’s all your Matt.

MATTHEW HEUSSER: Thank you, Michael for the fine introduction. We’re back, this time to talk about the tip of the spear, what is cutting edge in Agile testing. Where is Agile testing going? To do that I brought in two people that I think are absolutely positioned to speak to that. First we’ll talk about Jess. Now, I can’t remember where I met Jess… I know, it was in New York, at a testing meetup a couple of years ago, and since then, I know you’ve worked for bit.ly, and just a bunch of different companies doing API-like test automation, but tell us what you’ve been up to, Jess.

JESS INGRASSELLINO: Thanks, Matt, Yeah, actually we met at the QNA Meetup in 2013 in New York. So, in Midtown [laughter]. I actually just recently started at Salesforce dot org. Have been doing some automated testing, working very closely with dev ops to increase the availability and usefulness of test environments, and at Salesforce.org, even just in my first month it’s been a great experience getting to know the ops organization because they have a really amazing approach to test environments and dev-ops. That helps because their product is the most complex product that I’ve worked on.

MATTHEW HEUSSER: And that’s saying quite a bit. Welcome, Jess. We also have Garry Heon, who has been kicking around QA for a long time, including gigs as the Director of Quality Assurance for Eco-sys, and a partner at Meridian Precision, and Global QA Director at Expereon and continues his progression as a business manager for QualiTest group. Tell us a little bit about you, Garry and what you’ve been doing at QualiTest Group.

GARRY HEON: It just seems like yesterday I was helping companies adopt Agile methodology and now at QualiTest it’s… I’m helping companies take Agile and actually become a dev-ops shop, as our clients are looking to deliver products faster into marketplace and really integrate the development, QA and operations organizations together.

MATTHEW HEUSSER: Fantastic. So now you know a little bit about Jess and Garry, we should talk about the news, and I think the latest and most important and relevant news is this DDOS attack that took down everything from big time .com reliable websites that people were using to run their business to, like… I’m not clear on this it actually…

JUSTIN ROHRMAN: Spotify, Reddit, stuff like that.

MATTHEW HEUSSER: But it stole cycles from, like, your local DVR to do it, right?

MICHAEL LARSEN: That’s correct, yeah. So let’s set the stage for this. There’s an article that was just recently published on this called “Meet the Company Whose Webcams Shut Down the Internet Last Friday”, and yeah, in this case, what makes this interesting is that this absolutely massive DDOS attack was not perpetrated by bot-nets comprised of computers, but instead, from internet connected devices. Last Friday’s attack was carried out by compromised DVR’s and webcam devices infected by a piece of malware called Mirai. It’s not particularly sophisticated malware, but it’s proved to be very effective because, as was quoted in the article “Internet connected devices often have poor security and easy to guess default passwords.” Those of you who have your home routers, or if your web cam has a password that you’re not quite sure about, might be a good time to go change that.

MATTHEW HEUSSER: So is that like the default network name is “cisco123” and the default user ID password is “admin123” and somewhere there’s just a text file with a list of all the default usernames and passwords and whenever it gets into a network it just has to crack that, and whenever it finds a machine it just tries to crack that?

MICHAEL LARSEN: I would be impressed if it was that advanced, to be honest. Seriously it is disturbing, I’m able to… I take CalTrain, and I, just as a little experiment, I check out my little Wi-Fi list, and it’s amazing how many completely open and just ready to connect to devices I can find. I don’t, you know, because that’s just rude and wrong, but if somebody were less scrupulous, they could easily go in and jump on just about anybody’s machine and take over it. So the thought that you can do this with your DVR or with a webcam or with an Internet of Things device in your house… exciting times, yes?

MATTHEW HEUSSER: When I got my router out of the box, it had a username and password on it.

JUSTIN ROHRMAN: Did your Internet adapted toaster have good security on it? Or refrigerator?

MATTHEW HEUSSER: Yeah, probably not. Devices can be traced back to a single company, Hongxiao Jeungmei Technology. They said that weak security helped contribute to this attack. Specifically, it’s their circuit boards that were the vulnerability. It says that it’s patched the flaws with its products in September, 2015, and its devices now ask the customer to change default passwords when used for the first time, but older products are still vulnerable. So, you’re talking about devices that have been out there in the wild for the better part of a year, and more.

JUSTIN ROHRMAN: People probably don’t even know they own these devices, too. How many people look at the board inside of your web cam to see what manufacturer it was?

MATTHEW HEUSSER: Yeah, so in the good old days we would have a registration card that you would fill in and you would mail. And then the manufacturer would know that you bought it, and could send you a letter or something. People don’t really do that anymore.

MICHAEL LARSEN: No, it’s all online.

JUSTIN ROHRMAN: Yeah, and now that’s just a guarantee of extra email.

MATTHEW HEUSSER: Right, so why even bother? But to get into the DVR it would have to get through my router first. It would have to get through my cable modem. So there’s quite a bit of insecurity there, there’s multiple layers of error. That is the thing that nobody thought about. The Webcam manufacturer’s a US based company, bought a bunch of parts and stuck ‘em together, and they didn’t have anybody working on security of the board. So that problem killed ‘em. And I think that is dev-ops and Agile testing, high reliability, deploy often. So what I worry about is “what are we forgetting to think about?” When people talk about “I’m going to press a button and get a green bar, and we can deploy because I got a green bar”, I think about that Black Swan problem as a way that testers can add value. That’s a highly compressed explanation, but am I on the right track there, and what do you think is coming up next for Agile testing?

GARRY HEON: I think one thing that this attack definitely changed is the way the entire market thinks about security. I know when I go home and I look at my Internet capable Samsung refrigerator or my smart TV, I never think about security because I really don’t care if someone hacks into my refrigerator until we can see how it’s actually used as a weapon to actually create an attack on other sites that are more secure through Denial of Service. I think we’re going to start seeing even a shift from the market in what people demand of these products that we really don’t think need to be secure. That’s definitely going to drive all of the manufacturers of anything that’s Internet capable to do more testing. The test team itself is going to have to start thinking about those things to make their products more secure.

JESS INGRASSELLINO: In response to that I agree completely that it will, I hope, encourage manufacturers to add more testing and to encourage more robust approaches to testing these products. Something that I’ve seen shifting, and some of the work that I’ve been kind of hired to do in multiple places, is to take a look at the entire delivery pipeline, from start to finish, and say, “OK, what does quality look like in each step?” I think that the role of the tester is becoming more extended to come to terms with, actually, attempting to prevent issues from getting into the product in the first place, by discussing testability, discussing quality in the product creation process before it ever gets into a development sprint for code.

GARRY HEON: I think Jessica brings up a good point. Where dev-ops is really starting to play into this is dev-ops is about delivering very quickly. All these Internet-ready devices can get far more upgrades pushed down to them from the manufacturer. If we can get from development through QA and actually through operations extremely quickly, they can go and patch all these devices very quickly, and prevent future attacks from happening in a short period of time.

MATTHEW HEUSSER: Oh there’s so much there, Garry, I’m so glad you brought that up, and Jess, too. Abbie Bangser –we should have her on the show, she’s great, she works for ThoughtWorks- she actually has gone into organizations that want to do something like dev-ops, they have to design an A/B split framework, and they have to design a blue/green rollout framework, so they can slowly shut down boxes as they ramp up others, and then if something goes wrong, you can roll it back. They’re just like, “we’re just going to build it”. No. You need to test that. You need to think about what could go wrong, and then create scenarios and run them through. So she’s actually tested these dev-ops like infrastructures. Speaking of which, with Jess’s point about collaboration in preventing problems, historically, a lot of testers just don’t have a great deal of technical skill. They press buttons to make sure that the thing happens. Failure demand is so high that that actually has more value than their salary, but when you take those people and they say “what about security?” I’ve been in the room when testers say, “what’s going to happen when you’ve got the same user logged in on to different devices, two different browsers, and they try to do conflicting things?” This is during planning. The testers are told, “no user would ever do that, sit back down, the requirements are one device at a time”. Of course, they put the thing in production. Six months later, the call center is lit up, because a bunch of people are using their tablet and their computer at the same time, and it’s not working. I agree with you that that’s where we need to go, but the average tester is not there yet. So how do we get there?

JUSTIN ROHRMAN: In terms of a lot of the specialist aspects of testing like –security is an obvious example, and also performance and load testing- I don’t think these things even fit into a dev-ops cycle. Most dev-ops work is built around very granular functionality. The specialist testing jobs are generally done off to the side, outside of a development sprint, because somebody noticed a need. It’s not like a pull system there.

MICHAEL LARSEN: So I worked for ten years for Cisco Systems, and I got a chance to work with them from when they were a very small company with just a handful of products, to becoming a very large company with a lot of products, and an intersection of hardware and software… how do we look at Agile or dev-ops to include actual hard products like boards and circuits? I mean, that’s a bit out of the sphere of your typical “software as a service” model. When we think of Agile, when we think of dev-ops, yeah, we’re thinking of a web site and we’re thinking, “Oh, we can do a quick A/B test, and if we like this feature we can push it out”. Hey, if you’re designing a brand new switch fabric for a Catalyst switch or a Cisco router, and they have circuit boards and firmware and microcode, but you have to work with the intricacies of the actual hardware a lot of the time, and when you have to do a hardware push or a hardware tweak, that’s just not something that a dev-ops or an Agile environment is realistically going to answer… or am I wrong?

GARRY HEON: I think you bring up a great point because the push around dev-ops is a very high level of automation, from end to end in the process. So I think you raised a great question when it comes to, “How do you deal with the hardware itself?”

JESS INGRASSELLINO: I think that, if we extrapolate the abstract from the idea of dev-ops, which is “fast, reliable, repeatable, predictable”, these are things we want to have happen, and these are reasons why automation of all types has become the de-facto standard. What is to say that that kind of process oriented thinking can’t be applied to hardware testing? I say this having worked at Nomi years ago, and it was a company that was purchased by Brickstream, and I did hardware testing there. That hardware testing, of course, we had customers that wanted this hardware and needed it in their stores and needed upgrades. The focus was on breaking down what needed to be tested, then working out, hey, how are we going to test this? How are we going to work through the scenarios and address customer concern, address our concerns, because it was not all our hardware. It was third-party hardware with some of our software installed. I think for us to keep that within two week sprints, we did break it down and we did focus on developing some thought processes around being able to repeat what it was that we were doing and to be able to say, “OK, this is what normally happens, this is what we want to happen”, all of the typical kinds of questions that we would do with software, just adding the layer of complexity that we would need to use that with a hardware device that we did not necessarily have control over.

MATTHEW HEUSSER: Awesome! If we’re talking about the tip of the spear, the tip of leading bleeding, the embedded devices that update themselves for security so that you can do something like continuous delivery while the device still runs, or attempts to run, is certainly an area that needs research and work. I agree with Jess, it’s possible. Most of us aren’t Elon Musk’s, so most of u aren’t going to have your car say “please pull over, rebooting” [makes beeping noises]… and if you bingle the update, that’s really bad. Now you’re stranded in Nevada with a car that doesn’t run. Do today’s testers have the skills to contribute valuable things in that dev-ops environment, and maybe we should take a step back and define what we mean by “that dev-ops environment”. A short version of that would be “a technology infrastructure in place such that you could do continuous delivery if you wanted to.” Do today’s testers have those skills, and what needs to change to get them there?” I’m going to throw that to Garry.

GARRY HEON: So one of the changes that I’ve been seeing with our testers in a dev-ops world is, we no longer have these testers that are purely automation people and these other testers that are more thinking about the product and thinking about writing test cases and then manually executing them before throwing them over the wall to the automation people. What e are finding is we need one person that pretty much does it all. Here at QualiTest we’re calling that “QualiTesters”, so that’s our term for these people. What we’re doing is, in a one week sprint, we have the tester writing the test cases and writing the automation, so when the developer is finished with their development work, and they do the build, it deploys to the environment and the tests automatically kick off. So we are seeing a bit of a shift on the skill level that we really need more of these automation type folks as part of the team.

JUSTIN ROHRMAN: So, in my head what that sounds like is you want programmers instead of testers.

GARRY HEON: No, I want testers that are able to program.

MATTHEW HEUSSER: Justin, let’s be fair, isn’t that similar to a major project you’re on right now?

JUSTIN ROHRMAN: I suppose it is. Yeah, in a way the… well, we’re not doing continuous delivery, and the reasons that I’m doing high volume UI automation are different from the goal of small incremental updates. I joined this project after having ten plus years of testing experience and study, also. The demand I’m sensing in modern dev-ops shops is for programmers that can write lots and lots of extremely granular test code or, alternately, pairing to do that.

MATTHEW HEUSSER: When you say “granular”, do you mean API level tests?

JUSTIN ROHRMAN: API or lower, yeah.

MICHAEL LARSEN: I’m also “Living la Vida dev-ops”, so to speak, in my own environment, as I’m a software tester, I’m also the release manager, and I do a lot of manipulation with AMI’s and with Amazon Web services, and being able to make sure that boxes come up. So I totally understand the idea that… it’s not that you need programmers, per se. You need people who can test, but who can also write code to be able to implement those tests, and there is a fair amount of odd granularity that you have to be able to play with, and be able to explore and emphasize that into code, and sometimes it takes a number of iterations to get right. That’s definitely been my experience on a regular basis. Our team, just because of economics and people moving onto different projects, we’ve really had no choice; we’ve had to become a much more expansive group, where full stack developers and full stack testers -is there such a thing as a full stack tester?- but, where we have a lot of interaction and play with each other in regards to where it’s not at all unusual for me to say “hey, I’m going in and I’m finding this problem with a log”, and instead of just saying “oh, can somebody go take a look at that?”, it’s “oh, I know where that file is, let me go take a look and see… oh hey, guess what? We have a parameter for that that we’re overflowing”. Not just be able to say “hey I found a problem”, it’s also “and I can recommend the fix for it, or if you want, I can even try the fix out myself and verify that it works, or I can push that myself.” There’s a whole battery of scripts that I have been working on, before we get them into Jenkins, that I have to run myself, and I’ve gotten pretty good at making bash scripts that will allow me to go in and set up machines, that will allow me to go in and set up security profiles. And then I port them into our system, and they become just as much code as any software developers code. So we’re definitely seeing a blurring as to “am I a tester? Am I a programmer? Am I a build master? Actually, I’m all of the above and it fluid.

GARRY HEON: I think… you said it in there and it’s a very subtle point that, maybe, people didn’t hear, one of the things we have to be very careful about when we start looking at the testers of the future is “yeah, there’s this heavy development focus on what the testers need to do, but first and foremost we have to make sure that person has the tester mindset. If we focus on the development aspect and not the tester mindset aspect, we run the risk of actually creating poorer tests with less coverage, because the thought process is a little bit different.

JESS INGRASSELLINO: I’m gonna’ jump in and say to Michael that I love the concept of the “full stack tester”. You might have coined a term there. I have been, actually, to talk a little bit about my latest role at salesforce.org, I’m there working with Chris McMahon and he’s built a good portion of the automated UI suite already, and at my previous jobs I built the automated UI suites for bit.ly, for Nomi, and I did the front end for Rent the Runway, o I’m familiar with these technical aspects. The reason I feel like “wow, full stack tester. I could relate to that concept.” I think, first and foremost, I do have what would consider a tester mindset, and in fact at SalesForce.org I was hired to “shift left”, to shift testing left and make sure that quality is included as early on as it can be included in the process, and then also, my technical knowledge makes it easier for me to communicate with people about what’s going on and makes it easier for me to pair, so even at a larger company like SalesForce.org, I was still hired primarily to do the exploratory, conceptual testing, with the knowledge that I would be pairing and working with dev-ops. Absolutely I think that all of those things, especially the mindset, are critical in a tester, and my technical knowledge has definitely helped. As things move more towards a dev-ops culture, it’s something that people who have that knowledge, they’re going to find that their lives are easier, I guess, because they don’t have to push themselves to learn all of it, so having it is certainly a help. The mindset piece is absolutely, as Garry said, that’s critical. We don’t want to lose that. I find that the most brilliant testers that I’ve had the opportunity to talk to and work with, that’s their defining feature. Everything else they’re able to learn as they need to.

MATTHEW HEUSSER: Let’s talk for a minute about… we jump from the “I Google for testing level obvious stuff” but we get really deep and we talk about the social dynamics that happen on projects. Michael and I have this experience at Socialtext that has happened so many times. When I was there, they hired Rick (Scott), they hired Chris (McMahon), and they hired me. Each time they said “we’re going to get a programmer, we’re going to get an automator, we’re going to get someone to come in and look at our back log of test automation that was too hard to build, and that person is then going to build that infrastructure out”. Same thing happened every single time; Rick jumps in, he has to both keep up with the pace of the ongoing development and build this infrastructure out, so he starts out with “I’m going to be a programmer, I’m going to do network automation stuff…” and a month later he’s a tester. He’s just doing testing. He uses the framework to the extent that is possible, he doesn’t extend it. They bring in Chris, same thing happens. Chris adds a CI framework in Ruby because he’s kind of a Ruby guy, but he doesn’t extend the Perl, because he’s not a Perl programmer. I come in, I’m a Perl programmer, and I do a little… and I mean a little. I actually made a few commits that were “wrap a for loop around this command line parameter so you can say “for everybody” and you wouldn’t have to type in the email addresses, ‘cause all I do is get the object and do a for loop around it.” Or, I write some automation around search and tag, little things like that so we’re not cut and pasting six lines of code but instead calling a function. We all get sucked into testing. So Garry mentioned this sort of “both”; I think it’s extremely challenging to have… well, there’s the maintaining the infrastructure, and there’s writing the automation that keeps pace. Let’s be honest. When you’re writing the automation that keeps pace, and you have to keep up, and there’s some risk that is outside of the capability of the framework, like “printing looks bad”. What are you going to do when printing looks bad? You’re going to have to export to PDF, which Selenium can’t do, because it can’t drive the menus. You’re going to have to create a PDF, and you gotta’ look at it to see if it looks right. Defining “looks right” is extremely challenging, and some awesome programmer in the back is gonna’ say “well… you could pull out all of the fiddly bits that are the date logic, replace them with today’s date and time, or just remove them completely, and then have a static file that has the dates removed, and then you can diff them. It’s always possible -maybe this is beyond the scope today- it’d always possible to write that kind of automation. Nobody ever does it. So we do it manually and hope it doesn’t happen again.

JUSTIN ROHRMAN: So, if I had to summarize this, I’d say the discussion is, once again bringing it back to the tester role, where it fits in an organization, and is there room for it in companies that are trying to do dev-ops, we can claim that having full time automators is the same thing as having a tester, but I think it’s a fundamentally different problem. They’re thinking about building frameworks and tools and things of that nature and not… my opinion is that they are not so much thinking about actual testing of the product. The question is “where does the role of the tester go whenever we talk about moving to dev-ops or moving past Agile?”

GARRY HEON: I think as companies move more towards dev-ops, especially new companies that are just getting into it for the first time, they’re reading a lot of great things on the Internet about speed and automation. One of the things they need to keep in mind is the role of the tester, and what the tester needs to do doesn’t change. Although we’re going to automate maybe and push towards that automation, we still need to understand the fundamental role of the tester and what needs to be accomplished, right from unit testing through the end to end to end testing.

JESS INGRASSELLINO: as I’m thinking about this, I’m thinking about the testers, and I would encourage new testers to inform themselves about all of the different ways that testing is “performed” in different organizations. Especially ways that they may not be comfortable with or familiar with. So if they’ve learned about exploratory testing, I would say they should take a look at automated testing. If they’ve only ever written automation frameworks, I’d say “OK, go look at exploratory testing”. I would encourage them to look at some security testing tools. Encourage them to understand what Jenkins is, just so they can say they have some understanding of a deployment framework. I’d say, for testers, go educate yourselves, especially in territory where you feel a little bit unsafe, so that way you’re prepared for this brave new world of dev-ops and the tester, because it’s not going away. That’s the one thing that I’ve seen.

MATTHEW HEUSSER: And on that… what are you up to now? Where can people go to learn more about you and what you do? What’s coming up soon?

JESS INGRASSELLINO: Anybody who’s interested in my random test and other thoughts can go to my blog. It’s jessingrasselino.com. Anyone who’s interested in teaching Python –it’s something near and dear to my heart- teachcode.org is my business. I work with New York City public school teachers, and I’ve written a Python book, so I’m very much invested in the Python education community, and will be chairing the PyCon Education Summit this next year, and he call for papers is going out in a couple of weeks.

MATTHEW HEUSSER: As for me, I’m going to be coming to Germany for Agile Testing Days in December, and we’re bringing that to the United States, AgileTestingDays.us, in June. That’s going to be in Boston, North Shore area. We just ramped down a ton of work, and we’re just starting to ramp up a ton of work. It also looks like I’m going to be in Agile and Beyond in May, but I’m hoping next year to not overdo it on the conference circuit, and actually get some work done and see my family a little bit.

JUSTIN ROHRMAN: I’m gonna’ be at home for a little bit, at least ‘til the early part of next year. I have a baby due on November 7^th, so any day now basically. So that’s going to keep me away from conferences.

JESS INGRASSELLINO: Hey, congrats!

JUSTIN ROHRMAN: Thank you!

MATTHEW HEUSSER: Our sponsor is QualiTest, it’s the only reason we can do this. It’s a lot of work to get this together. It’s a lot of research and time and investment, and Qualitest does underwrite us and provide us some funding. Which is kind of a big deal; we don’t talk enough about that on the show. We try really hard to make it non-commercial, but I think Garry should close us out with what’s going on at QualiTest.

GARRY HEON: So, we’re going to be in Austin, Texas and Atlanta, Georgia, for a couple of CIO summits. We’re going to get the opportunity to meet with the leadership teams of some of the major corporations in the United States, as well as technology leaders within the financial institutions, so that’s our big upcoming happenings in November.

MATTHEW HEUSSER: Thanks, everybody, for coming.

JESS INGRASSELLINO: Thanks.

GARRY HEON: Thank you.

MICHAEL LARSEN: Thanks for having us.

JUSTIN ROHRMAN: See you.