The Testing Show: Twenty Years of Qualitest with Ayal Zylberman

]

Panelists:

References:

• Equifax Credit-Card-Data Breach Could Be Largest in U.S. History
• Year 2000 Problem
• Mercury Interactive
• Pacific NW Software Quality Conference
• Senses Working Overtime: Improving Software Quality Through Accessibility and Inclusive Design
• TestBash Philadelphia 2017 I Live!

Transcript:

MICHAEL LARSEN: What you are about to hear is a special episode of The Testing Show and also a deviation in our planned show order. Our previous episode Number 43 is Part One of a two-parter on Machine Learning with Peter Varhol. This episode is a twentieth anniversary episode for QualiTest as a company, and we recorded an interview with QualiTest CEO Ayal Zylberman to celebrate the occasion. That is what is going up as Episode 44. Episode 45 will be the conclusion of our talk on Machine Learning with Peter Varhol and that will be posted in two weeks. With that, on with the show.

[show intro]

MICHAEL LARSEN:  Hello and welcome to The Testing Show. I’m Michael Larsen, your show producer and today we have our regular guests. Jess Ingrassellino?

JESSICA INGRASSELLINO:  Hey, there!

MICHAEL LARSEN: Perze Ababa?

PERZE ABABA: Hello, everyone!

MICHAEL LARSEN: Justin Rohrman?

JR: Good morning.

MICHAEL LARSEN:  Our moderator, Matt Heusser?

MATTHEW HEUSSER: Hey, hey!

MICHAEL LARSEN:  And we’d like to welcome Ayal Zylberman?

AYAL ZYLBERMAN:  Hello, everyone!

MATTHEW HEUSSER:  Ayal is the CEO of QualiTest. Not QualiTest USA, not QualiTest North America but the international company based in Israel. We’ve got the top dog. And he founded QualiTest twenty years ago. The company is doing its twentieth anniversary celebration. So Ayal, that’s a terrible introduction. Tell us more about your background and the company and how you got started and what you were doing before QualiTest.

AYAL ZYLBERMAN:   Wow! You’re talking a long time ago. As you mentioned, QualiTest was founded in Israel, which is the country I was born and raised. As an Israeli, I had to spend three years in the Israeli Army. First two years I was a combat soldier but in the last year I was assigned to supervise the implementation of a new artillery system in the Israeli Army. Actually, I was doing testing but it was not called testing back then. We’re talking about twenty-six, twenty-seven years ago. When I finished my military service I was called by an Israeli military contractor who got a contract to build an artillery system for the Swiss Army and they asked me to properly test the system. Before that, they didn’t have any testing at all so I went there. I used my military background as an artillery soldier and I was testing the system and made sure it works properly. I spent about two or three years doing that, then the system was finally launched, quite successfully. Then I joined another test consultancy company called TestCom, which don’t exist anymore. That was already the year 1996, when the old millennium bug just started and all of the testing companies were very much focused on testing or helping companies to prepare themselves for the millennium bug and the whole catastrophe it was planned to be on January 1, 2000. I decided that’s not something I’m interested and I knew quite sure when it would all be ending, so I decided I want to start my own company and focus on what was then called “traditional testing”. That’s how I founded QualiTest.

MATTHEW HEUSSER: Thanks for telling us about your background. Before we get to the deep interview, I want to cover news really quick and that’s… we had the Equifax hack last week, which seems to be they just didn’t keep their servers up and patched. Someone was running a continuous loop attack against all of the computers on the Internet and found a vulnerability. They hacked half of the personal credit information of Americans in the United States. So overnight, somebody in my family’s Citibank card was hacked, by which I mean someone made a purchase on WalMart.com. Not the first time this has happened. Called Citibank, was on hold for three minutes, and then someone in a very poor accent said, “Citibank’s systems were down, call back in two hours.” Then, clicking the logoff, yielded a Citibank.com error message. So, is it possible that so many accounts were stolen and then hacked that we’re basically seeing a denial of service against Citibank because of the sheer number of support requests? Or is something else going on? This is prediction time. By the time this show comes out, we’ll know what really happened. So, this is prediction time. If anybody has a thought… or is the Internet just held together with duct tape and baling wire and we can just expect more of these? What’s going on?

MICHAEL LARSEN: Well, if you have to ask if the Internet is held together with duct tape and baling wire, you don’t know much about the Internet. Of course that’s the case [laughter]. I’m sorry, having worked for Cisco for ten years back in the ‘90s that’s just an expected response. What I found interesting with the whole Equifax issue is that they’ve known about this for awhile. They announced it after about a month of trying to figure out what was going on with it. What I think we’re dealing with is a bit of fall out. There are certain people who are responding who are trying to reactively set things right on their end or what they think they’re setting right on their end. It’s entirely possible that new system details or things having to do with trying to take care of people’s accounts are coming to bear and it’s just frustrating everybody involved. As far as a prediction is concerned, I have a feeling that a lot of people are trying to put in new systems to cover their butts and it’s causing delays and issues. That’s what I’m really curious about is to see if this is something that’s going to come back where the cure is worse than the problem.

MATTHEW HEUSSER: We’ll find out. Anyone else have thoughts?

AYAL ZYLBERMAN:  I have a slightly different angle to view this and I think that is the quality perception. I think what I have noticed in the last five to ten years is that the whole quality perspective has changed. Ten to fifteen years ago, quality systems were systems that had no bugs or a minimum amount of bugs in it. The perception? Equifax will never be able to launch any quality application from now on because they will have this security thing and having vulnerable systems means low quality and that is the perception of the users. If there is one thing that we as testers can learn from Equifax, we are in the business of ensuring quality. Quality is not about having the system running totally, it’s also about targeting of vulnerability of systems. If ten months ago, in order to maintain your online bank account you had to change your password every month, that would be considered as a low quality system for you. Nowadays, if the system doesn’t force you to do that at least once a month, you don’t feel safe about them. So I think there’s a great impact on the quality perception and that is only just one example.

MATTHEW HEUSSER: That’s fascinating. We no longer trust the very systems we’re using. Because of that we expect to have to change our passwords because we expect to get compromised and hacked. That’s terrible. So let’s dive into the interview now and we sort of asked already how QualiTest got started. The comment about security and the perception of security gets me wondering, those first few years, what were they like? What kind of projects were you doing? What was being tested? What was the test strategy that you were using in 1997 when you founded QualiTest?

AYAL ZYLBERMAN:  Wow, that’s a great question. I think testing was very different. In the old way it was perceived by the market testing was not a necessity. It is now a necessity. All companies, every team that develops software require testing or a certain kind of quality assurance. Back then most of the systems that were developed were either tested by the developers themselves or not tested at all. When we started QualiTest, it was very much like any other startup. We didn’t have any offices so we used a coffee shop, which was our official office (we even got letters sent there) and we tried to pitch the market to invest in testing, so it was not very much around why to engage with QualiTest, the selling pitch was “why is testing needed?” Once we were able to convince the customers that they need to test their software, we were not the only option but there weren’t any options back then. So that is one aspect. First few years we were mainly around gaining more and more clients from different industries and different verticals. I founded the company together with Eli Mogolin (sp?) so we started to company together, the two of us, and we did a lot of consulting work for clients. The first two and a half years I was engaged with one of our clients as a QA manager and later QA director, so I managed a team of about 40 testers working worldwide. Unfortunately, that company, TeleKnowledge, doesn’t exist any more but they were quite successful back then. One thing that really worked for our benefit and I mentioned that before is the old marked was very much focused on the Millennium Bug and we were focused on traditional testing. In 1998 we partnered with Mercury Interactive, now part of HP or HPE. Back then, Mercury was a very small startup company. When we first partnered with them they had about twenty people in total. I think they now have about 20,000. Actually, I was one of the first users to use Mercury tools in one of my projects. That’s it. We’ve grown by hiring the right people and by getting more and more clients and more and more testers working with us and professionalize what they were doing.

MATTHEW HEUSSER: And how large is QualiTest now?

AYAL ZYLBERMAN:  Currently right now, we have roughly 3,500 testers worldwide. Our main business is in the U.S., so that’s more than 50% of our total business but we are also quite strong in Europe and Israel, which is our home country.

JESSICA INGRASSELLINO:  I love that you got started in a coffee shop and you were literally receiving letters there, I think that’s fantastic and very hopeful for people who want to get started. The thing to do is to “get started!” How over time has the clientele for QualiTest changed and the kind of problems that you are seeing now twenty years later? How are you addressing them?

AYAL ZYLBERMAN:  Obviously, a lot has changed. I’ll start with the staff angle. Back in 1997 when we hired testers I think the main problem that we were facing was convincing people to treat   stuff. 63% of our total operation is SLA based engagements. It’s a fixed price engagement. We are being measured not by the quality of the people but the quality of the deliverables and I think there’s been great progress in terms of having metrics and KPIs. Twenty years ago if you were to ask CIOs or VPs of R&Ds whether they are happy with the quality of the product, you will get mixed verbal answers. Nowadays, if you ask the same questions you get numbers and you get KPIs and you get metrics. Yeah, we mange to find 94% of defects and we had only 6% escaped defects. That is a quality metric being used more often than it used to be twenty years ago.

MICHAEL LARSEN: So you’ve been around now for awhile and you’ve had a chance to see some changes in the market and what people expect of a company that does testing. What do you think makes for an ideal client for QualiTest at this point in time? If you’re going to be working with someone for the long term, how do you interact with and how do you prefer to work with these companies?

AYAL ZYLBERMAN:  One thing we found out is that in order to make testing successful, you need to partner with your client. In the last few years we realized that the goal of having quality processes and quality systems is common t both us and the client and whenever the client perceives us as being a partner for quality, so being a partner, that would be my answer.

PERZE ABABA: I want to step back a little bit. You did mention that since you guys started, you have a little over 3500 employees. As a manager right now one of the biggest challenges right now is how to deal with growth and how to nurture that growth. There’s definitely some challenges to make sure that the growth becomes sustainable. How did you do that in QualiTest and what were your challenges? The strategies you employed in order to maintain your growth, did that have something to do with how software development and technology has changed and has your client base changed since day zero now that you’re on your twentieth year?

AYAL ZYLBERMAN:  I’ll start with the last one.  Yeah, I think our client base has changed. When we started QualiTest, we were very much focused on technology companies. Nowadays, we work mainly with large enterprises, mainly banking sector, health care, life science. We now focus on enterprise and large companies. From a technology perspective yes, a lot has changed. If you were testing a banking system you could take a bank teller and make them a tester because then you would get someone that really understands the business processes, procedures and basically the business ecosystem. Nowadays we see testers become more technical of a job. If you can’t code, you probably can’t be a tester today. There’s a new term I recently heard, a “full stack tester”. Like full stack developers, if you take test automation as an example, ten years ago, there were manual testers and automated testers and manual testers couldn’t write any test automation scripts and test automation experts knew nothing about the business processes and the business ecosystems. Nowadays, those two roles have merged. Companies expect testers to both understand business logic but also be able to address the code, both to write automation test scripts but to also to run some DevOps activities; unit testing, integration testing and so on.  I think what supported the growth of QuliTest –this has always been our strategy- is to bring managers in place… not senior management, we’re talking about mid-level management, what we call business managers. There are currently four in the hierarchy of QualiTest. We have myself as a CEO, MDs, Division Managers and Business Managers, and still we call our Business Managers “Mini CEO’s” because they get a lot of responsibility that also comes with great accountability. Yes, we have 3500 people but they are all being divided into small units and each unit is being operated individually. You need to trust your managers and you need to provide them with the responsibility required for them. You need to make sure that you bring people that are accountable and are well linked to the values of the company and you also need to give them the freedom and to trust them and be tolerant to any mistakes they might do (and they will do a lot of mistakes) along the way. Second thing I would say that we did that was quite unique was many companies are separating the technical responsibility from the financial responsibility. So you have different people, if you look at QA managers, most of the QA managers, they are not the one who make the decision regarding what kind of people they want to hire, how much money are they willing to spend, those kind of decisions. So there is one person who is making the financial decisions and one person who is taking the financial responsibility. What we did was merge those into one person. Quality and money goes hand in hand with each other. You can always spend less money and obviously you’ll get less quality and the decision on how much quality you will eventually achieve is a financial decision. It’s not a technical decision. What we did, using the same Business Manager or Mini CEO structure that I described, the Business manager has both financial and technical responsibility and I think merging that is something that I would advise any company either any IT services or Enterprise company to use. Did I answer your question?

PERZE ABABA: Yes, very much so, thank you.

JR: Hi, Ayal. QualiTest has an interesting position in the market in the sense that you all have a lot of market share, so you have the ability to influence not only how companies perceive quality and also what the average of software testing looks like from a practitioner point of view, so I was wondering if QualiTest has any projects? What sort of things are you all doing to shift the market just a little bit, if you can even talk about that?

AYAL ZYLBERMAN:  I’ll just mention two initiatives. One is more concrete and the other is something that we are working on but that’s more of a future thing. The current initiative is mainly around digital transformation. There are many companies that are going through huge change transferring what they do manually to a digital way of doing stuff. It’s a lot about using mobile applications and having technology in place but the real digital transformation is around the mindset. Both internally, companies that are leveraging more and more technology and change their old ongoing activities and externally the way that the customers or the end user are engaging with those enterprises. We’ve create a set of tools and a set of solutions that are all around ensuring proper digital transformation. Part of that we have coined a new term called “business assurance”, which is very much different from quality assurance because, if you look at the current technologies, there’s a lot of artificial intelligence, data science, machine learning, neural networks, all kinds of new platforms that change not only the way that systems behave but also the way you test those systems. In the past, when you tested a system, you compared that system to the requirements but nowadays, you and me will open Google and type “Barcelona”.  I’m a great soccer fan, so probably Google will auto complete “Barcelona Football Club” while you probably interested in Googling Barcelona because you want to be a tourist there and go there, so it probably will auto complete about the best places in Barcelona. So you can’t really compare the results of your test to requirements because using Artificial Intelligence provides different kinds of results for different kind of users. So the old testing has changed from that perspective and that’s part of the digital transformation initiative that we have been implementing for the past few years.  So we are now using more crowd testing. We are now using more heuristic based testing as a test approach. We are using some tools from security point of view but there’s also other stuff that influence quality. The second initiative is a future one, so we don’t have that working but we are spending a lot of time and a lot of effort developing that and this is what we l the “test bot”. Usually when people speak about test automation, they don’t really mean “test automation”. What they really mean is “test execution automation”. So all the automation that relates to testing nowadays is only around execution of test cases. If you look at testing from a broader perspective, testing has to do a lot more than just executing. You need to learn the system, you need to plan your test, you need to analyze the results, you need to chase the bugs, and there are a lot of things that are around test automation. We believe that, in the future, there will be more parts of the tester’s job that are being automated. If there’s one thing that we’ve already managed to automate, it’s the test selection, which is, in my view, the most unstructured phase of testing. Usually you let your testers or your test team leaders select which test cases you want to execute on each iteration or each release. You can have a machine making that decision in a much better way because the machine can take into consideration all the changes that were made in the code. We have tools in place today (Jenkins, GitHub) that can give you very detailed change analyzers in terms of code and a machine can know which test should be executed to cover all the changes that were made on the code. You can log into a trouble ticket system, bug systems, previous execution system and you can take all of that information, put it in one system that can select the test cases that should be executed better than a tester. We believe that in the future there will be a technology in place that will be able to not only select those tests but technology that can generate test cases that can analyze the results , even if you have test automation and test execution automation in place, you still need to have a human being analyzing the results. In the future, you will have systems that can analyze the results and, if needed, report bugs and defects. So this is the future. It doesn’t mean that we will not be needed anymore as testers, it will only mean that our job will significantly change.

MATTHEW HEUSSER: Great, thanks! We actually just finished up a couple of rounds of conversation about the value of machine learning and AI and its applicability. I appreciate the safety language around there. “we believe that in the future,” meaning we could be wrong and we don’t have it yet. There has been some marketing saying, “right now you can if you just buy our magical tool,” and we’ve been a little skeptical of those claims. Last question I want to ask, you said that you were working with Mercury when they were twenty people and then they got really big. I remember looking at them when they had just piles and piles of job openings in Silicon Valley. Then they were acquired by HP and the tool became “Universal Functional Tool” and then that UFT and QualiCenter kind of became increasingly welded together -not that you couldn’t use them separately but if you used them together they worked together well-  and thenHP has been sold to MicroFocus and there’s been a lot of criticism, skepticism that with the MicroFocus merger, MicroFocus does not do a great job in supporting most of their products. Most of their products they kind of sunset, keep them on life support and collect annual fees for maintenance. Do you have any insights or thoughts about this MicroFocus deal? Is it going to be good for customers? Should people be looking around at other products/ What’s this going to mean?

AYAL ZYLBERMAN:  I think the MicroFocus merger is the result of a trend that started a couple of years ago that went way beyond testing and that is that there is a great change nowadays of moving from commercial tools to open source tools. In the past, open source tools were mainly focused on small companies, startup companies, and nowadays you see large enterprises that are using open source tools and they get more confident out of that. Looking specifically in the testing market, I think that Selenium became much more popular in the last few years. Five years ago many testers haven’t heard about Selenium and now it has become something that exists in most of the organizations I visit. There’s also Bugzilla and there’s also some other test management tools that exist. Obviously I don’t have any insights on the financial results of HP or HPE but I would assume they’ve had a certain decline in sales. Obviously I’m not smart enough to make a decision or to make a view on that. I’m sure the HP guys know how to make their own decision, but I think that commercial tools become less relevant nowadays and the market is definitely going to more open source tools.

MATTHEW HEUSSER:  Any final thoughts before we let you go?

AYAL ZYLBERMAN:  Nope. I think we had a very interesting discussion.

MATTHEW HEUSSER: The mail segment for the show is, if you want to talk to us, you have a question, you have a comment, you have a guest to recommend, you want to be on the show, email address for the show is  thetestinshow (at) qualitestgroup (dot) com. This is the part of the show where we do announcement. I talk a lot, everybody knows what I’m doing. Michael? Perze? Jess? Justin? What do you have going on that we haven’t covered lately?

MICHAEL LARSEN: This show is scheduled to air at the end of September, so for me, I’m heading into the final couple of weeks before the Pacific Northwest Software Quality Conference (PNSQC) in Portland, Oregon. I’m doing a talk on Accessibility and Inclusive Design and interesting enough, even though I am focusing on a number of testing topics, I’m actually part of the Development track this year. So that’s October 9-11 in Portland. Beyond that, I’m shifting into end of the year mode and looking forward to what I hope will be a relatively quiet winter.

MATTHEW HEUSSER:  Great! Anyone else? Jess always has stuff going on.

JESSICA INGRASSELLINO: I do always have stuff going on. As ever, keeping up with writing a lot of new articles, so keep a lookout for those. The test conference coming up at the end of November which is in NYC . We’ll give a plug for that. Giving a few presentations and a talk there.

MATTHEW HEUSSER:  It’s looking like I might come to the Pacific Northwest in November/December possibly. I don’t know where; Spokane, Portland, so if you are in that area and you wanted to get in touch with me or have me give a talk o something at your local users group, give me a note. I’ll definitely drop by Salem, Oregon at the very least.

PERZE ABABA:  I might be going to TestBash in Philly first week in November. Still waiting on final approvals for that but I do look forward to meeting up with everyone else and seeing what’s happening in that space. Philadelphia is actually pretty exciting. Recently, mainly because of the exponential growth in startups. It’s been a very interesting space considering we only have this type of situation either in New York or Boston or of course on the West Coast but never in Philly so I guess this is a good time to be a tester in Philadelphia.

MATTHEW HEUSSER:  And when is that?

PERZE ABABA: I believe it’s in the first week of November.

MATTHEW HEUSSER: Our friend of the show Tony Gutierrez who I’m not sure has been on The Testing Show is going to be speaking there so looking forward to it. All right, I think that’s a wrap then. I’ll let Ayal have the last word and we’ll call it a day.

AYAL ZYLBERMAN:  Just want to thank you guys for hosting me. It was a pleasure.

MICHAEL LARSEN: Thanks for being on the show.

MATTHEW HEUSSER: Thank you.

JR: Thanks.

PERZE ABABA: Thank you.

JESSICA INGRASSELLINO: Thank you.

[End of transcript]