Insights Podcasts The Testing Show: SAP Testing

The Testing Show: SAP Testing

May 16, 2019

The Secret Weapon of QA Modernization

SAP (Systems Applications and Products in Data Processing) is a large suite of software products best described as being an Enterprise Resource Planning (ERP) system.

Many large scale companies use SAP to cover a variety of processes related to accounting, finance and commerce, as well as a variety of other functions. While a tester may not be working directly with it, it’s a very good bet that if you visit sites that are large enough, you are absolutely interacting with a SAP system(s) somewhere.

In this episode, Matthew Heusser and Michael Larsen welcome Simon Evans to talk about all things SAP, the ways in which it shows up in so many places, and ways that testers can level up and come to grips with this large ecosystem of inter-related systems.













Michael Larsen: Hello everybody and welcome to The Testing Show. It is May 2019. We are joined by our ever present MC, Mr. Matthew Heusser.

Matthew Heusser: Hello, good time zone.

Michael Larsen: And for this episode we are joined by Simon Evans and we’re going to be talking about a kind of special area of software testing. We’re going to be talking about SAP testing. Simon, how about you introduce yourself.

Simon Evans: Sure. Hi, everyone. I’m Simon Evans. I head up SAP and Enterprise Practice within Qualitest. My background is SAP through and through, so probably 20 odd years in that particular discipline. Started on the implementation side of SAP before moving into testing some time ago.

Michael Larsen: Fantastic. As is frequently the case on this show, I get to play the either newbie or ignorant one, I guess is the best term. I have somehow managed in the 30 years that I have been a software tester to never have to be involved with a SAP environment, so help me out here and probably those who are listening to this, possibly for the first time, saying, “SAP testing, what’s that?”

Simon Evans: That’s a very good question. SAP, it’s not one individual thing which is why it makes it very difficult to explain. It covers entire businesses across all the different industry sectors. If we kind of distill it down as simply as possible, pretty much all businesses who buy stuff, they make stuff, they sell stuff, and then they’ve got the finance, etc. That sits behind it. And it covers all of those areas, so it’s back office, it’s middle, and it’s also the front end user experience as well.

Simon Evans: So when we talk about testing there’s multi disciplines within it and the most important skill that our team carry is actually understanding businesses. When they go in and speak to a chocolate manufacturer, they understand how their business works versus a professional services firm like a law firm. They need to understand how they work. Mostly they’re very different business industries. Does that answer the question? It probably doesn’t.

Michael Larsen: It gets me closer. Is this an ERP system basically?

Simon Evans: It gets batched as ERP, so yes, is the simple answer, but there’s many other dimensions to it as well that fall outside of the ERP domain. There’s a whole digital platform around eCommerce elements. There’s big data analytics. There’s many different views around the SAP product sets. There’s hundreds of products within it.

Michael Larsen: Wow.

Matthew Heusser: When I talk about ERP systems and customization for testing, there’s two big pieces. We have a big system that has a bunch of databases, has a bunch of data it as you described. It has all our customers, all of our orders, maybe our manufacturing process hooks into that, maybe our HR process hooks into that, payroll, finance.

Matthew Heusser: Usually there are some funky custom business processes that are just special for us. So we have to add additional screens to kind of hold those data. We need to flow that into other things, like applying a discount or something. It’s very unique. It’s an old customer. It’s a strange contract.

Matthew Heusser: Then we’ve got to both test our custom code and test the upgrade, moving from version 4.7 to version 4.8, or 5.0. Then we’ve got to make sure that our custom software works with all of the changes to the back end database in SAP.

Matthew Heusser: When we talk about testing SAP, is that the kind of thing we’re talking about, or is there something else?

Simon Evans: That is what we’re talking about, but some of the challenges with it is there are lots and lots of different systems and not all of those will be SAP. There’s a diagram that I sometimes use to scare our sales team when I’m trying to explain SAP because it’s a third of what one of our customers is doing. There’s a SAP bit sitting in the middle, but then there are literally hundreds of other systems, things like zip code verification, credit card checking, all these different things that aren’t SAP. But we still have to include those which is why the business process is king in our world.

Simon Evans: Whether we’re actually testing an upgrade, or we’re testing a new implementation, or they’re just making changes, businesses never sleep. We’re having to follow those business processes and validate and manage the risk of these companies because that’s what the do day in, day out.

Simon Evans: They’re not using systems, they are running their business, whether it’s tools for HR or hire to retire activities, or it’s payroll, or manufacturing lines, production lines, et cetera. They are real businesses running, and we have to test and make sure that their businesses work, not the software works, that’s just a by product. We need to make sure that business works because if it doesn’t they lose significant amounts of money very, very quickly.

Simon Evans: The emphasis is always on, does the business run? And even when we get into performance, again, it’s about does the business at the speed that it needs, hitting the peaks that they have as a business, not is the technology good enough? We need that to work, but we need to make sure that the business is not impacted in anything that they do.

Michael Larsen: You mentioned another use case there. A couple of them. We’ve got these outside systems, or finance systems, or data warehouse, our order entry, our EDI, these sort of outside systems that have to sort of merge into ERP and then back out again. Then there’s the actual conversion to SAP from whatever we had before. It’s probably in some cases, 30, 40, 50, 60 different systems that are all sort of… the goal with ERP is to get them to converge into more like one, and you really end up with seven, but there’s one main one which is your SAP. Those all seem like valid SAP testing projects.

Michael Larsen: We talked about upgrades for a hot minute. SAP used to be a Windows based application where you’d actually run version 7.4. My understanding is it’s web based now. How do you track versions and how does testing work with software as a service?

Simon Evans: Well, this is another one of the challenges. Elements of SAP are web based, but not all of it. Different customers will do exactly the same thing, but some will do it web based, others will use traditional GUI view and there’s about six different interfaces into SAP that a user could use.

Simon Evans: When we’re doing the testing, again, if we go into two companies that are essentially identical. If you look at something like Mondolase or Cadbury’s with Nestle, they all make chocolate. The way that they go about doing it and those business processes that we refer to at the end, will be different. There will be some significant differences there. And how they interface with SAP and the various applications will be different.

Simon Evans: So, we have so many dimensions that we have to contend with when we’re testing, different technologies as well as different business processes, and how they go about doing things. It’s really that combination of things that makes these projects very, very complex. Which is why they tend to be very big projects and customers invest significant amounts of money even just to do upgrades which may sound simple, but the upgrades they’re going through at the moment, the transition onto S4, which is the new core platform, takes a minimum of a year, and many customers spend three or four years going through that upgrade process.

Michael Larsen: You mentioned six different entry points for it. There’s web based. There’s Windows based. There’s several of each. How does the way you test change based on how the customer uses SAP?

Simon Evans: In many respects it doesn’t change. It’s about testing the business. So, how the user interacts with the system to an extent, doesn’t really matter as long as it’s a good experience for them. We’re not necessarily too concerned about… obviously we want it to be a positive experience, but whether they’re using the GUI platform, or they’re coming via the web, or whether they’re coming from Citrix or anything like that, it’s just got to work for them as a business. It needs to not slow them down and it needs to functionally work to allow them to do their jobs. We’re always tackling it from the business process perspective and if it’s a webpage, great, it’s just a different way of entering in data.

Michael Larsen: Fair enough. I have a couple of questions here and I think maybe we can group this into one. Again, I’m coming at this from a complete position of being new to this. Let’s say that I as somebody who is an up and coming tester suddenly find myself in the guise of, “Hey, you’re going to be testing a SAP system.” Okay, I don’t even know where to start. What are the tools that are available to test a SAP system? What kind of resources are there for somebody like me to actually come up to speed on working with a SAP environment?

Simon Evans: The first port of call is not necessarily around tools, but it would be to say, “Okay, which part of the business am I working with? What do I actually need to understand? Are we talking about finance? Do I need to have some kind of accounting level or experience? Are we talking about a warehouse? How do people stack pallets and stick them onto trucks to send out to customers?” And really start to learn how those things will work.

Simon Evans: When it actually comes to getting hands on and doing the testing, there’s a number of tools out there. But we are limited, partly because of the different interfaces that are out there. There’s the use of open source tools is limited within the SAP space, so it typically tends to revolve around three, three and a half, tool sets out there, depending on what you’re doing.

Simon Evans: And Microfocus products, they’re the predominant suite of tools that customers use, so the likes of ALM and UFT, Loadrunner, but then you have other areas of automation. There’s Worksoft. They’re a pretty big player certainly in the US around test automation. And all of these have to go across multiple different platforms, not just those SAP systems as well.

Simon Evans: And then the third one is Tricentis, and whilst they’ve been around for many years, they’ve kind of had a real big push into the SAP space. They’re the ones that are really pushing Microfocus because they’ve got more of a comprehensive platform, not just do test automation around the whole test management piece.

Simon Evans: And then the final one is The Half, which SAP themselves has some proprietary tools that sit within one of their areas called Solution Manager, and they’re there, they work, but it’s a bit like stepping back in time sort of 20 years. So, very few customers use those and also it’s pretty limited to within the SAP suite. If we’re talking about those interface systems again, you know, a third party logistics system, their tools don’t really work. So, it always brings you back to those first three that I mentioned.

Michael Larsen: So, there’s all these tools. They do test automation. In my experience, in practice, they run from a desktop. They’re not hooked into a continuous integration server. You’ve got to set up your environment so that you can run them. Do you actually see a material benefit to the customer? Do the upgrades go more smoothly? Do they put everything in place for version 4.7, and then go to 4.8, what’s the savings that companies experience from using these tools realistically long term?

Simon Evans: Realistically, long term, certainly around test automation, it’s big because any of the test scenarios that you’re doing tend to follow full end to end business processes. They take someone or sets of people a long time to step through it. If you think about your own interaction with companies that you buy from. If you order something from Apple, you might go onto their website, choosing what I want, choosing what color it is, all the different specs, and actually pay them some money. But what happens in the background, that goes off, the order gets placed somewhere, someone has to process that, ultimately, although they’re not making to your order, but a phone has got to be build somewhere along the lines. There’s all those processes. And that product needs to get shipped to you, et cetera. It’s like a long process and obviously once that phone has been shipped there’s all the aftercare stuff et cetera.

Simon Evans: So when we’re testing, we have to follow those processes and it takes time. If we’ve got that automated we can skip through those no matter if it’s seconds or a couple of minutes, as opposed to it taking… some of these tests take over a day to get through. The savings are huge from that perspective.

Simon Evans: One tool that I didn’t mention, the big impact area for us, where we’re helping customers, is identifying their risk. There’s so much… pick your favorite company, think about what they actually do, there is so many different dimensions. You can’t test everything. It would take you forever. So, a big part of what we have to do is going and saying, “What is the risk based on the changes that you’re making? Where do we need to focus? How do we make sure that if things go wrong your business still runs? Traditionally, we’d look at things like risk based testing, and we’ve got various different models that we kind of use with customers with the test predictor that Kay and I have been developing. Whilst that’s not SAP ready yet, we’re looking to build our methodology and our knowledge around risk into that particular product, so that we can look at these kind of changes and go, “Right. Let’s run some analysis. We’ve got a wealth of data there. Where do we need to focus? Which rocks do we need to lift up to find where these issues might lie and be very, very targeted? When we do that we tend to strip out a good 25% of time from a big program. By a big program that would be a minimum of a 12 month project that’s running, sometimes it’s multi years. You’re talking about months and months worth of saving for a customer which not only benefits in terms of deploying and realizing the business benefits sooner, but just the cost of those projects month on month. It ends up saving them running into millions of dollars.

Michael Larsen: Did I hear 25%? Was that the number that I heard?

Simon Evans: You did, yeah.

Michael Larsen: Okay. That seems reasonable. That seems realistic for a technology upgrade or conversion project. I can see that, maybe more the upgrade side than the conversion side.

Michael Larsen: Yeah. All right. Thank you.

Matthew Heusser: Some people won’t get to a number. They’ll wave their arms. Their numbers will be funny. They’ll be, “Well, if you’ve run the automation every single night, you’re saving so many tens of thousands of dollars versus having a human run it. “It’s like that’s not really… no, it’s not comparable.

Simon Evans: Time is a big thing for us which is partly why we have a dedicated practice as well because the skill sets are different. Our guys, their process experts. We have people that were accountants that are looking at finance for us and things like that. Because they make a real difference in knowing exactly what to target. So we’ve got some good case studies and metrics where we’ve gone in and replaced one of the big testing vendors, like a WhipPro or a Cognicent, et cetera. Where we’ve gone and we’ve looked at the amount the customer is spending on testing and the amount of time that it’s taking and we’ve taken things down with the metrics that the customer’s produced. It’s not even us fudging the figures. Where they’re spending around 25% of their budget on testing and we get that down to 10, 11, 12%. It’s a pretty huge saving in terms of cost. But then when you look at the time saving that we bring to them, that’s the real thing that they need. It’s all about their business. They’re only making these changes… They’re only going to go through an upgrade… They’re only going to implement something if it benefits them as a business. If it gives them competitive advantage. So, the sooner that we can get them across the finishing line, the sooner they achieve that benefit and that competitive advantage.

Michael Larsen: I think that’s a fair point because one thing I don’t hear about in the technology upgrade or the conversion space is that you’re basically running in place for six months, to a year and a half, to two years, and the entire focus of your IT department, really 90% of your focus, is on the upgrade. So you’re not building out new features, you’re not developing new capabilities, you’re not doing new and innovative things with your marketing, which means your competitors could be jumping ahead. When you shorten that window by six months that means you get six more months of productive actual developing new features and new value add. That’s something I don’t hear a lot when people talk about these sort of things. Just kind of, “Oh, we have to do this technology conversion. It’s the cost of doing business.” Well, what is the opportunity cost of that production upgrade, or that conversion?

Michael Larsen: Now right now you are at the SAP event for the year, Sapphire Now. Thank you for finding some time for us. You’re in Orlando, Florida, right?

Simon Evans: Yeah, we certainly are. You mentioned it being SAP’s flagship event. They have a few events around her but this is the big one. This is largely attended by key stakeholders, CIO’s, et cetera. It’s huge. It’s a big event with people attending fro all over the world. It’s a key one for us.

Michael Larsen: Now, I know our audience is keen to hear about how Britney Spear’s is singing, or whatever, but what’s new and hot at Sapphire now on the technology side? What is the newest announcement, the newest features, the newest… what’s under the covers that is finally being let out that you can talk about?

Simon Evans: Cool. There’s a few things that they’re focused in on right now. There’s a lot of talk… I mentioned S4 which is their new platform earlier on, which isn’t really a new announcement. These programs are so big it takes customers years to be able to make the step forward. There’s a lot of chatter around how they’re going to do that transition and lots of customers are making that leap, so they’re talking about how do we accelerate it, how do we manage the risk? If we’re changing all our platforms, if we can’t impact our business?

Simon Evans: And one of the nice things for us is there’s a customer being featured at the event quite heavily. Their CEO, CIO, have all been in attendance and presenting, and they’re a customer of ours. Tapestry, which is a brand house that owns Coach, Kate Spade, and Stewart Wiseman, made the leap early and we’ve been supporting through the testing process and helping them manage that risk through their transformation. That’s a great one for us and to see everyone’s talking about those.

Simon Evans: The other big talk is really around the customer experience. The way SAP look at the customer experience is it’s not just the front facing piece, the good looking digital front face of businesses. What they’re looking is that whole engagement. It’s the supply chain that sits behind it. It’s every facet of the experience that the customer will experience, not just when they’re kind of clicking on a website and interacting. It’s what happens through that whole life cycle and really what they’re honing in on is the amount of data points that you can collect for your customers in all different areas of the business and what we do with that data.

Simon Evans: So if you look at the big success stories over the last ten or so years, your Googles, your Apples, all the social media platforms, what is it that they’re actually doing? They’re being very successful with the new black gold which is data. And SAP are trying to hook into the data that’s existing within all of these organizations and transforming back into connective advantage which is what their escort platform does.

Simon Evans: That’s where all the talk is about is how to mine all this amazing data that these companies have and turn that into something meaningful.

Michael Larsen: That’s huge, I think. When I was in graduate school almost 20 years ago, talking about ERP conversions, there were three things that they said the typical ERP conversion was something like 225% over budget, something like 150% late. They’ve gotten better at that in the last 20 years. But also that ERP really hadn’t delivered on it’s promise to kind of turn everything into magical rainbows. What it had done is allowed companies to get a single unified picture of their data, and the next thing was going to be data mining. We may finally with machine learning be getting to a point where that’s practical for companies without hiring an entire army of machine learning experts at a high fee. That’s exciting. I should add, it’s Lady Gaga this year at Sapphire?

Simon Evans: It is Lady Gaga, this evening actually. I think she’s playing this evening.

Michael Larsen: I know that you are at Sapphire now. You’ve been gracious to take some time to talk with us and to find a quiet place to record with us too which I’m sure must have been a feat. Just so that we can get you back to your conference and what you’re doing, tell us a little bit about what you’re up to and how people can get a hold of you if they want to know more about this whole SAP testing world. Heck, I might be sending you an email because I’m curious.

Simon Evans: Well, I intend to be traveling around from country to country a lot in supporting our sales teams, so I’m always happy to drop in and have a conversation if people want to. The easiest way to get a hold of me is probably on Teams, or even just good old fashioned email. Either myself or my team, we’re always more than happy to talk, and support, and if you’ve got customers that you’re working with, and you just need a little bit of information to maybe peak their interest, do get in contact. There’s a stat somewhere that SAP talk about that 78% of all transactions worldwide touch an SAP system, so companies that you’re engaged with, companies that you even just go in and use… I mentioned Apple earlier in the conversation… If you go into an Apple store, you’re working with someone to buy something or they’re giving you support, what you may not be aware of is they’re directing using an SAP system. So it is absolutely everywhere and it’s almost guaranteed to be somewhere within your customers. We’d love to talk to you and support you through any conversations you’re having with them.

Michael Larsen: Fantastic. Okay. Well, at that point I think we have reached a good spot where we can put a pin in this and say thank you very much, Simon, for joining us and to everybody listening we will definitely look forward to seeing you in our next episode.

Michael Larsen: Thank you and be careful out there.

Simon Evans: Thanks for having me.

Matthew Heusser: Thanks a lot Simon.