The Testing Show: External vs. Internal Software Testing

Software fills many niches and does many things for individuals and businesses. Some software is designed and developed for mass user interaction, while other applications are designed for specific purposes and markets. Some software is literally developed to work within a company for the benefit of its staff and is never seen outside of the company it serves. Still, it feels as though the testing advice that is given and received often focuses on big players and customer facing software.

Panelists:

References:

• Digital Subscriptions Boost New York Times Revenue And Profits
• Infrastructure As Code
• 10 Lessons from Hertz’s $32M Web Design Lawsuit Against Accenture
• Black Box Software Testing Courses

Transcript:

Michael Larsen (00:00):
Hello everybody. And welcome to The Testing Show. As of recording, I will say today it is Wednesday, November 11th. We are in autumn. Some places seem to have skipped autumn and gone straight to winter. So our condolences, if that’s you or hey, if that’s what you want, awesome. In any event, I just wanted to say welcome to the show. Glad to have you here. And this is neat because for those of you who’ve been listening for a while… and when I say a while, I mean, five years, you probably remember that one of our regular hosts was on the show and he’s back today. So Perze Ababa, Hey, how you doing, dude? Welcome back.

Perze Ababa (00:38):
Hey, thank you so much for having me. It’s been a pleasure. I’m still a big fan of the show, of course. I continue to look forward to hearing from the great minds that are part and parcel of the show so, Hey everyone.

Michael Larsen (00:52):
And we’re glad to have you in the hot seat this time, because you are a guest of honor for our topic today. With that, you all know me, I’m Michael Larsen. Hey! And of course, you all know Matt and Matt tends to run the show here. So I’m going to let him do that. Matt, Take it.

Matthew Heusser (01:06):
Thanks, Michael. So if you don’t know Perze, he’s an established long member of the test community. We’ve run into him at conferences five, 10 years ago. And he’s got a storied background. I know that you were at the New York Times when they did their digital transformation initiative. Now you’re at Johnson and Johnson. Of course, you probably won’t be talking about any specific company, but you’ve been testing for… what were you doing before the New York Times?

Perze Ababa (01:35):
I was actually a testing consultant prior to this. Worked with real estate companies, insurance companies, travel companies prior to that. Primarily building, helping build software and then providing testing support.

Matthew Heusser (01:50):
Yeah, you’re on the customer advisory board for SauceLabs, for AppliTools. You were at Viacom. You were at NBC Universal, where you were a Director of Quality Assurance. So don’t sell yourself short. You’ve been around. The reason we wanted to bring you in was we’re having this conversation. A lot of the advice for how to do software testing is based on how we cut physical disks, CDs and floppy disks in the 1990s. That’s where a lot of the advice comes from. To the extent that we’ve moved past that it’s mostly from software companies. It’s mostly from Facebook, Twitter, Microsoft, Google, maybe Amazon. So the sort of big five software companies influenced the way we talk about testing. Most of everybody else is not working at a software company. Is using software as an enabler for some other business function. It’s been my experience that the techniques that would work in one environment might not work in another or missing context. So today what I’d like to do, if it makes sense is talk about when we started this discussion, what is a software company versus a non software company.And once we’ve narrowed that definition down, how would the techniques for testing be different if you were at Napa Auto Parts versus you were at eBay,

Perze Ababa (03:17):
I’ll go straight off the bat and go ahead and challenge the existing definition that you have of the companies you’ve called out. Facebook, Google, they’re known for their technical acumen, but at the end of the day, I think like Facebook, for example, it’s a social company. Their primary product is dealing with consumer data and they’ve built a robust infrastructure around making sure that they handle the data pretty well. And they gather insights and provide whether it’s, you can call it a better experience or something otherwise to their users, but they are focused on the social part. Of course, there’s some analytical parts there that’s introduced from time to time. It’s the same exact thing for Google. For the longest time, Google was a search company that has this really awesome infrastructure and algorithm that can discover things and what to do with these discoveries. They kind of shifted towards becoming more of a marketing company or like an ad selling company. They didn’t set up saying, “Hey, I’m going to build software”. Now, if you compare that with some of the vendor partners I’ve worked with, we do have really common companies. If we remember Computer Associates back in its heyday, they are purely a software company. I think aside from being a hardware company, Dell was also a software company at one point in time. So you do have these consulting firms that don’t seem to have a product leaning approach to things, but they would help people out in building those. And I think this is what you referring to Matt, where the Napa Auto Works and all the other, probably even mom and pop shops that need help with their technical domain. These are things that a lot of the consulting firms are helping with.

Matthew Heusser (04:59):
I think there’s brick and mortar companies like Napa or whatever that use the software as a channel to sell their stuff that have to upgrade their ERP every three years. And they’re going to bring in somebody like Accenture, maybe Qualitest. They’re going to bring someone in for a short period of time to do some testing and some development so they can get back to the business of selling auto parts, right? They’re in the auto parts sale business. I would disagree. I think the Facebook people knew they were writing software. They’re not selling software as a business model like Microsoft sells Windows, but I think they are software companies. And I really like your perspective on, are companies like Forbes, The New York Times, The Wall Street Journal. I don’t know how much of that revenue is physical paper anymore and how much of it is people subscribe to websites? Are they transitioning to becoming software companies and does the dynamic change, how are those dynamics different? So I asked about eight questions there. Michael, do you want to chime in?

Michael Larsen (06:03):
I can, actually. Let’s look at this. Having worked at various companies that have tried to sell software that people use everywhere from anybody who offers an app or anybody who offers a product. I worked for companies like Connectix, which was the Virtual PC company up until they were acquired by Microsoft and also worked for a company that made Immigration Law software, which was one of my most interesting testing realities ever. I was working on that environment and that was neat because I got a chance to see what the difference was between say you’re selling a software service and that service is something that allows you to interact with whatever process you need to do. Because of the fact that we had software that was designed for immigration attorneys and their paralegals, and being able to process that type of stuff, that was a very different experience working on that, then it was, say, working on software like Virtual PC, what type of things we focused on and what we cared about. Virtual PC, of course, you have your literal virtual machine environment and there’s a lot of stuff that you can look at and a lot of technical details there. But once you get the abstraction thing down, a lot of it flowed pretty clearly. And those things didn’t change very often. Legal stuff or having to update forms or having to modify the things related to I-9 documents or I-131, or fill in the blank, all sorts of things that… I’ve kind of blocked because I worked with it for so long (laughter). Just you really got into some interesting details there that would be very different than say, testing a software that sold something. This is the neat thing and maybe Perze I should probably kick this over to you at this point, because I think that we as testers look at the products that we work with or that we test based on who they serve. If we’re serving a particular niche customer, we’re going to focus on those niche things that really matter to them. And there are other things that we might go, “Oh, usability is not so great, but they don’t care because it’s not really the most important part of their workflow”. Whereas if you’re making something that’s more of a customer-based or online e-commerce type of a site, you really do have to care what the customers are because they’re not a niche thing at all. It’s an “everybody”. And so if you’re testing a product that a broad section of people are going to use, you need to pay a whole lot more attention to things than something that’s going to be used by a very niche community doesn’t. Does that make sense?

Michael Larsen (08:45):
Definitely. Michael, I mean, I’ve definitely noticed varying degrees of the types of the systems that we’re building. You mentioned about the customers that we had to be dealing with, whether it’s somebody that’s inside the company that you’re working for, or pretty much a customer that you’re working with, like John Q Public or John Doe. The differences for these is really how are you actually building it? Are all the IP of the software that you’re working with. Is it all built in-house or are you doing a ton of integrations? Or is it purely integration where you’re just using someone else’s APIs or libraries and you’re extending that? Is it a lot easier to test? Eeh, it depends because the more dependencies that we have outside of things that we can’t control, there’s definitely more surprises that will come in, especially when there’s changes. Whether those are changes in your data, changes on the rules that you’ve built in, as well as the infrastructure that supports it. It’s definitely a challenge in a way that if your organization doesn’t have a good grasp of why are we actually doing this, or are we just doing this for the sake of being in with the times, then we should take a step back. And I answer those fundamental questions and define the scopes of what we’re building and how we’re building will just follow.

Matthew Heusser (10:06):
Yeah. What you said reminded me of a couple of things. I think if you’re building software for YouTube, it’s not fun. It’s not pleasant. The switching costs to go to Vimeo, it’s nothing. It’s CTRL-T and then type in vimeo.com and hit Enter. Whereas if you’re writing internal software for the call center, or you’ve only got 200 call center reps, so you can’t scale it out the way you can scale out costs among 500 million Facebook users. And they’ve got to use it. If there’s a bug, man, click the back arrow and re-enter it correctly. You’ve got a stupid error? Well, why’d you enter the bad data, figure it out. We’re paying you a living wage to use this software. It doesn’t have to be perfect. The standard for good enough might be lower, so much more if you’re doing software by spec for internal. So it just has to meet the spec. When I was working on commercial software at Socialtext, people were much more open to this idea of, “This feature makes me feel bad about myself. Can we please fix it? [laughter] Please improve the user interface. Here are some things that bug me”, that were accepted and became minor feature requests, as opposed to when I was working on contracted software, whatever, it was, “Well, this is what the spec says. Just make it fit the spec”. Probably the most painful experiences of my career were in government compliance, where we didn’t even understand what the documents were saying. We just needed to come up with some explanation for why what we were doing was complying. We couldn’t even talk to a person. I went to my project manager once with a document and, “What does this mean?” And she was like, “I don’t know, just make it comply”. So all those forces, I think drive commercial software to be, for lack of a better term, to more surprise and delight the user enterprise software. And if you’re working in a company are probably working on enterprise software.

Michael Larsen (11:56):
If I can, I’d like to throw in one here too. So in the gaming world, I worked for Konami Digital Entertainment back in the early 2000s for a couple of years. And that was at the time when the PlayStation 2 and the original Xbox and, I guess, the Nintendo GameCube were kind of the Kings of the Hill. In other words, every game that you played was based on an actual physical medium disc, it was a CD-ROM or DVD or one of those physical mediums. You didn’t have a hard disc, or if you did have a hard disc, it was very secondary to store data or maybe to copy a game over. But the point is everybody got their games on those CDs. What that meant was if you shipped a bug, that bug was eternal, it was not going to get fixed. Period. So there was a lot of aggressive, “Oh my gosh, we’ve got to make sure that we get this thing right”. Or as right as you can, within the scope of gameplay and what that means. And one of the things that I was kind of proud of was I developed a system of forensic grounds related to certain titles that we shipped. We realized, “Oh, there was a bug. Oh, okay, well, how do we get around it? How can we warn people ahead of time?” And we made this database of forensic workarounds for, “Hey, if you find yourself in this given system, we know that there is a bug here and we know that there’s a problem here. We want to alert you to it before you even get there and you can avoid it.” Now, with the more modern systems and the fact that I think a lot of games are now delivered digitally, you can still buy a game, but you can also update the games which you couldn’t before. That’s not such a terrifying prospect. You can fix things on the fly, whereas just 15-some-odd years ago, you couldn’t. And I think that’s made a change, especially from that perspective. So it depends also if you’re going to run into a bug, how permanent is it?

Perze Ababa (13:49):
That’s definitely something that’s changed over the years. And I think it has also something to do with how the architecture of software has been defined and redefined over the years, especially with the advent of the more modern systems that has to deal with build and deploy. I do remember when we still used to ship CDs when I worked for an insurance company. Whenever there’s a change, whether it’s just a minor change, like your configuration value change, this is something that’s really “sent”. It gets very expensive if you do that. Right now, for example, the advent of infrastructure as code there’s things that we can easily switch here and there from a flag perspective. Of course, we have to tell our customers that we’re doing this, but it’s definitely simplified a lot because now there’s these moving parts that we can test in isolation and see how it affects the larger picture. So aside from the newer technologies that enable us to deploy things on the fly safely, there’s definitely a lot of help with the understanding that we have and how we can design our systems.

Matthew Heusser (14:54):
Yeah. I think the two mistakes that I see are a company that makes things, they go to the web, they try to treat it like an internal project. And the software isn’t high quality based on what I said earlier, or maybe they pay way too much. Was it Hertz, paid way too much to work with, I think it was Accenture. I think this is public. I think this is a settled lawsuit, so I can talk about it in this way. They paid way too much to get software that wasn’t very good and they didn’t know how to manage it because it didn’t have anybody in the room who had managed a publicly facing software project before. That’s problem number one. Problem number two is you’re really working on internal call center software that is not going to scale out to that many people. And you get your advice from Google or Facebook on how to do testing. And it’s just not relevant. You don’t have the same kind of problems. You don’t have the same kind of customers. You don’t have the same kind of budgets. You don’t have the same kind of technical staff, frankly. Those are, I think, the two ways you can fall down, they’re both problems of ignoring contexts and I’ve seen them both. So do you agree those are pitfalls and how do we stay out of it?

Michael Larsen (16:05):
I think one of the biggest pitfalls that would happen when we farm out building of software is that if we give them full control, we get very waterfall with the expectations that we have. Especially when we don’t manage them. It becomes a problem. I know you brought up Hertz, that’s an example for this. Then I was Googling some details about that. It’s just a website redesign their cost of $32 million. And I guess they weren’t happy about it, but I’m very surprised why it’s a surprise to them that they weren’t happy about it. It’s like if we’re working an iterative approach to doing this, defining proper parafunctional requirements, especially from a performance perspective or a usability perspective, there’s going to be that continuous feedback that we give them on whether things are still okay or not. All of these “requirements” should never be set in stone. These are guidelines on how we think things should work, but you know, our most important person that makes that decision on whether this is acceptable or not has to provide feedback, as early as you can, and as often as you can.

Matthew Heusser (17:14):
So to keep this out of… It’s a fascinating story we could take in a lot of directions. That software perspective on this is build it iteratively, have working software periodically, and have some external group doing the auditing of, “Is that actual thing fit for purpose?” I think both of those and company building internal software has the tools to use. And if not, they could rent it from somebody else. For a project that big not having somebody guarding the hen house seems kind of like a mistake.

Perze Ababa (17:48):
Right! And I think that continuous measurement of what is valuable is definitely there. A lot of the tools available for us, a lot of them are very technical. You measure coverage, branch and line statements, but it doesn’t really provide feedback on, “How does this current solution that we have relate to the people that we’re solving these problems for?” And having some sort of a continuous, even if it’s a surrogate measurement of how these things look as your delivery progresses, That’s definitely key. And these are things that need to be defined up front. I would hope that the people that you’ve hired to do these things are accountable enough to bring this front and center for you. Because as a result, this would provide more work, but you’re more honest with what you’re delivering. And all of that, the tooling should serve the greater purpose, which is, “Did I actually solve the problem of my customer that I was hired for?”

Matthew Heusser (18:43):
It reminds me of… Years and years ago, I worked with an insurance company and we were going to integrate with a banking software. So you had your health savings account. Could you go and swipe your card? And it could go to us. We could process the claim and we could say the claim is, I don’t know, $500 or something, like you’re going to owe $500. And then either we send you a check or we pay it out of your health savings account. So you paid with your credit card and you’re going to get a check in the mail seven days later to cover your credit card bill. Boom! I remember going to several -everyone who would listen to me- and I would say, “This thing is supposed to be live in a month and a half. We’re supposed to be in the middle of testing. We haven’t gotten it to pay one claim end to end yet. The end to end flow of, I put a claim in the system. It processes, it goes into the health banking system. It looks at your bank account and sees that you’ve got enough money and it cuts you a check. I haven’t ever seen us do that once with one claim yet.” Nobody seemed to understand what I was saying. We delayed the project by three months, one month at a time, and then canceled it. Made some saber rattling about suing the vendor. And the whole thing just went away. It was like a $10 million project because nobody ever tested the dang thing end to end. I think it’s just the basics. I think it’s getting through the whole system and seeing if it can work and then questioning it. And I think that’s the big failure. It’s the number one thing to do to resolve this problem of companies that have internal software that needs to do an external project, then want to test it like it’s an outside system. That’s the biggest failure that I’ve seen there?

Michael Larsen (20:24):
I would say, “Yeah”. Most of the products that I have personally worked on have been related to individual users or a very specific need. One of the more interesting products that I worked on. And I frequently come back to this one, mainly because of the fact that it was a niche industry. It was based on immigration law. I oftentimes like to talk about that experience because it was really unique and it had some very… esoteric challenges, let’s put it that way, that other products that I’d worked on, you would never have to consider. We would have to say, “We are going to place less emphasis on whether or not the product is super smooth and usable in a certain way” to, “Is it doing the right thing from a legal perspective?” That was much more important than whether or not the screens flowed perfectly. And going back back to your earlier comment, we’re paying you to use this software inside of a company. I’d say… I don’t want to pick on them but they’re a very big company. I think they can handle it. Let’s say Salesforce. You use Salesforce and you are actively working with it. Well, my guess is there’s probably some things in Salesforce that’ll drive you up a wall. You’re probably not crazy about their design, but you can deal with it. It’s okay. There’s a lot of investment in it. People are involved in it. It covers a whole lot of ground and makes a whole lot of money when all is said and done. So, you know, some little niggling things might slip through and not be considered that big of a deal. Now, if you can’t actively do your job or you can’t actively complete the work you need to do, that’s a much bigger issue. On consumer side products that let’s talk about an app or let’s talk about something that people just download because they want to use it. What’s the barrier? There isn’t one. If I decided that, “Hey, I downloaded this app. Oh, I’m not really enjoying this app”. Delete! I’ll go find something else. Even if I’m paying for apps, I’ve done that too. This app’s going to cost me 20 bucks. Okay. That’s a niche thing. I could certainly give it a try. I do a fair amount of music production. I’m a singer in a band. So I download and I use software that I actively pay for. In that regard, I’m not going to waste my time on a piece of software that is super fiddly and that I have to tweak with, that I can’t quite get to work right. Or it doesn’t feel good, or I can’t actively work it into my flow. If I have to radically change what I’m doing, naah, next! Move on. I’m not even going to deal with it. And if I lose… I’ve even spent $50 on a plugin for my recording software. And I’ve just decided, “Nah, it’s not worth it. I’m not even going to bother with this.”

Matthew Heusser (23:12):
Sure. You’ve lost that 50 bucks and they made one sale, but they lost all of the recommendations for all of the people that you might’ve said to buy it. How many times have I recommended QuickBooks online to people?

Michael Larsen (23:26):
Oh yeah. That’s why I use it [laughter]. And again, I’m not going to call anybody out on this regard because I realize when you’ve got a boatload of choices and you’ve got other options and the barrier to entry or to adoption is low, then you’ve really got to pay attention to if your software makes sense, if it works, and it’s comfortable. You can’t just say, “Well, it’s a good application. It does some really cool things but if it’s got some headaches, who cares?” A lot of people will care. I’ll use recording software as an example. For a couple of decades, ProTools was the standard and they knew it. You had all sorts of headaches and frustrations, but you know what, if you want it to be viable, and if you want to do anything with electronic music, you use ProTools and I’m not criticizing ProTools. I’m just saying they, because a bunch of other manufacturers stepped up their game and made the barrier to entry so much lower, ProTools is now no longer the corner of the market. And they’re more responsive because of it.

Perze Ababa (24:34):
Piggyback to what Michael was saying is there’s a tool that’s very near and dear to all of our hearts, AKA testers. Everybody has heard and used, potentially, HP Quality Center, which became HP ALM. That’s kind of where they fell, too, where people who now have 20 years worth of data, that’s stored in their infrastructure is having such a hard time migrating out of that tool into the newer more effective ones. But you just have to be able to bite that bullet. Again, going back to taking responsibility of asking the question, what are we actually using this for? Breaking that apart into workable chunks? And then you can perform your migration, give your bosses some sort of a timeline saying we can do this within two years, but we want to make sure that as we integrate and move people from one tool to the other, we made sure we have a pretty strong support system that gives them the ability to jump from one system to the next.

Matthew Heusser (25:33):
Before we go. Thank you, Perze. So I’ve been using Intuit QuickBooks for 10 years now. I pay for the privilege. It’s good software. And I actually met John Roberto who was working for Intuit at the time. So I think it really makes a difference if you make good software, if it’s external and knowing which context you’re in matters. And thank you, guys, for playing along with me with this thought experiment that is going to help influence my work and was super not baked when I came on this call. So thank you for helping me sharpen it. I think it’s that time for the show where we talk about what we’re doing next though. I know Michael, you’ve got a bunch of stuff going on. I I’m vegetating out in my house and turning into a potato, but tell us what you’re up to.

Michael Larsen (26:21):
Okay. Well actually, I’m kind of excited about this. We’re at the end of conference season. So there isn’t anything going on. I’m really hoping that we can kind of get the COVID situation under control because seasonally, it looks like we’re getting snow in the Sierra and I really want to go snowboarding this year. So I’m hoping that that’s going to actually come to fruition. Yes, that’s a literal knock on wood, but one of the things that I am doing and I had stepped back from for a while, but I’m getting back into it is I am doing some more external writing and I’m getting some stuff published in places I haven’t done before. So a few things should be coming out soon on that front. I won’t announce anything exactly until I see them actually fall, but yeah. Expect to see my name on a byline in a few more places. So, yay! I’m kind of excited about that.

Matthew Heusser (27:11):
Sweet. Perze? I know that you do a good job balancing family and work, and we’re honored that you found time for us. Are you coming up for air anytime soon? Where can we see or run into you?

Perze Ababa (27:22):
Well, thank you for that, Matt. I, I do appreciate it. I don’t know if I’ve quite achieved that balance yet, but I’m definitely working hard at it, but all my kids are schooling remote. So that’s really my primary focus at this point. My work has graciously allowed me to continue to work from home for the foreseeable future. So that’s something that I’ve been able to balance effectively. I’ve actually tried my hand again at helping teach BBST lessons for the Association for Software Testing. And it’s been a while and I was like, I thought I could deal with it for an extra hour or a couple of hours on a daily basis to facilitate the class. And I find myself very challenged with balancing that time and getting back into the saddle for teaching BBST has been a new source of joy for me. And I do look forward to helping assist teachers in the nearby future. And this is something that I want to be better at because of “time that I have on my hands”. So if you want to say hi to me, you can say hi to me directly on Twitter, but you could see me as an assistant instructor for BBST classes, with the Association for Software Testing.

Michael Larsen (28:33):
And those of us who also teach those classes appreciate that. All right. So I guess that brings us to the end of our show. Thank you everybody for joining us. Thank you Perze, for playing along with us. It’s good to have you back on with us and hopefully we’ll get a chance to talk to you again sometime soon. And for everybody else, we look forward to seeing you on a future show. Thanks for listening. Take care everybody.

Matthew Heusser (28:55):
Bye everybody.

Perze Ababa (28:56):
Thanks everyone.