A primary concern in any industry is the return of an investment (ROI). The ROI represents the benefits gained from the investment versus the cost that was expended. For each decision that managers make, they should be considering the ROI, and collaborating with the customer to ensure that everyone agrees upon options that maximize the ROI. While there is an obvious expenditure to implementing a new process for a team, the benefits are more difficult to calculate.
One such software testing process that has become popular recently is the Scrum method, which is meant to increase a project’s ROI. To evaluate if this approach is correct for your software testing project, we must determine how to effectively compare the different methods, such as Waterfall vs. Agile, and establish which will gain a better ROI.
The issue is that it is difficult to compare development methods. Numerous existent studies use pseudo-science and estimation to evaluate each method, where the conclusions are based on how the method fits into Agile principles or how the companies “feel” that the method is working out for them. In one such study, project managers were asked to compare their experience with popular processes, and a conclusion was based on the popular choice. In other studies, metrics have been developed to estimate the costs of these methods and perceived benefits. For example, one study calculated costs and benefits by the lines of code that were produced. However, there is no guarantee that a greater number of produced lines increased the effectiveness of the team, and the same method may not work for your company or your project. The field of software development process evaluation needs to be advanced in a more empirical and studious manner.
Establishing Methods
While a methodology may work for one company, one size does not fit all. You should start your development process by evaluating different methods and assessing the gains for your own business. The first step in comparing methods is to establish a scenario where two or more processes can be evaluated through the same guidelines. This can be achieved in two ways.
The first approach entails comparing two projects that are similar in size, scope, and cost, then developing each project using a different process. However, this approach does not guarantee success, as two projects are never exactly the same, and it can also be difficult to transition a team from a traditional testing methodology to an Agile methodology.
A better option is to take a large project and break each process into pieces, allowing you to compare the processes together. You should perform the testing processes against the Traditional methodologies, and then transition the team into the Scrum methodology by performing the same processes agilely. While this approach does have some of the same pitfalls as the first approach, the risks will be mitigated by the number of pieces and by the transitory approach. The “piece approach” could take a couple of iterations, due to your team adjusting to the Scrum process.
Hard Calculations for ROI
After the test scenario is set up, hard values should be established to calculate ROI. One measurement is the time required to establish a stable build. While this definition may differ per company or project, a good measurement of a stable build is one that contains no critical bugs. By measuring and calculating the man hours used, we have a hard value of which method costs less.
Another method is the number of critical bugs found, which cost considerable resources to correct. By correcting the bug in both methods, you can measure the resources used in both methods and compare the expense of both, then compare any savings to the amount of money invested, thus calculating your ROI.
Scrum and other Agile methods boast certain benefits that are hard to measure empirically. How can we concretely measure the consequential flexibility of a method? One option is to analyze how test scenarios deal with added features and measure the time for incorporation. Scrum boasts a collaborative process that increases communication between your employees. You could track communication errors through emails or other modes and see how many mistakes were made. Another principle of Agile is customer involvement; the primary goal of this is to increase customer satisfaction. We could survey the customers after the projects are delivered to see which process they felt was better. Furthermore, some employees could work better under different processes, and their opinions should be taken into consideration. However, these benefits are difficult to empirically measure and thus are not easily added into the ROI.
Decide the Method for YOU
There are many proposed ways to speculate ROI. Studies may suggest methods for your company’s project based on size or scope, but these proposals will never be as effective as testing the process in your environment and measuring your own results. Scrum and other Agile processes have great values and intentions, stressing business value and expediently delivering a better product, but that still does not mean that these are the best practices for you, your company or your project. Also remember that when learning a new methodology, a team’s initial attempt may not be representative of the best work that can be output with that methodology. It is important to find a method that fits you and your team.
