Insights Blog Evaluating the ROI of Scrum

White Paper

Evaluating the ROI of Scrum

As Scrum is the most popular framework adopted by organization adopting an Agile approach for project management, many companies are trying to find financial facts that justify its adoption. This article discusses the topic of evaluating the return on investment (ROI) of using Scrum and suggests some hints about mistakes to avoid and on how to get meaningful results from this activity.

Neri Lavi

Many strategies are supported as the “best” management of a software development process. While each programmer has his favorite way to code and every manager has his own method of administration, a software company’s goal is to always find the most efficient process to maximize profits.

A primary concern in any industry is the rate of return of an investment, or Return on Investment (ROI). The ROI represents the benefits gained from the investment versus the cost that was expended. For each decision that managers make, they should be considering the ROI, and collaborating with the customer to ensure that everyone agrees upon options that maximize the ROI. Any backlog should add value to the project. While it clearly costs to implement a new process for a team, the benefits are more difficult to calculate.

One such process that has become popular recently is the Scrum method. Scrum’s main purpose is to increase the entire project’s ROI. To choose the approach that is correct for the project, we must determine a method to effectively compare the different methods and establish which will gain a better ROI. However, an issue exists in comparing development methods. Numerous existent studies use pseudo-science and estimation to evaluate each method, where the conclusions are based on how the method fits into Agile principles or how the companies feel that the method is working out for them. In one such study, project managers were asked to compare their experience with popular processes, and a conclusion was based on the popular choice. This is bad science. In other studies, metrics have been developed to estimate the costs of these methods and perceived benefits. For example, one study calculated costs and benefits by the lines of code that were produced. However, there is no guarantee that a greater number of produced lines increased the effectiveness of the team, and the same method may not work for your company or your project. The field of software development process evaluation needs to be advanced in a more empirical and studious manner.

While a methodology may work for one company, one size does not fit all. You should start your development process by evaluating different methods and assessing the gains for your own business. The first step in comparing methods is to establish a scenario where two or more processes can be evaluated through the same guidelines. This can be achieved in two ways. The first approach entails comparing two projects that are similar in size, scope, and cost, then developing each project using a different process. However, this approach does not guarantee success, as two projects are never exactly the same, and it can also be difficult to transition a team from a traditional testing methodology to an Agile methodology. A better option is to take a large project and break each process into a small, medium and large piece, allowing you to compare the processes together. You should perform the testing processes against the Traditional methodologies, and then transition the team into the Scrum methodology by performing the same processes agilely. While this approach does have some of the same pitfalls as two projects, the risks will be mitigated by the number of pieces and by the transitory approach. The “piece approach” could take a couple of iterations, due to your team getting used to the Scrum process.

After the test scenario is set up, we need to measure hard values that can be used to calculate a ROI. One such measurement is the time required to establish a stable build. While this definition may be different per company or project, a good measurement of a stable build is one that contains no critical bugs. By taking this measurement and calculating the man hours used, we can have a hard value of which method costs less. Another method is the number of critical bugs found, which cost considerable resources to correct. By correcting the bug in both methods, you can measure the resources used in both methods and compare the expense of both. By measuring these values, you can compare any savings to the amount of money invested, thus calculating your ROI.

It is important to note that Scrum and other Agile methods boast certain benefits that are hard to measure empirically. While Agile methods are implemented due to their flexibility, how can we concretely measure the consequential flexibility of a method? One option is to analyze how our test scenarios deal with added features and measure the time it takes to incorporate them. Scrum boasts a collaborative process that increases communication between your employees. You could track communication errors through emails or other modes and see how many mistakes were made. Another principle of Agile is customer involvement; the primary goal of this is to increase customer satisfaction. We could survey the customers after the projects are delivered to see which process they felt was better. Furthermore, some employees could work better under different processes, and their opinions should be taken into consideration.  However, these benefits are difficult to empirically measure and thus are not easily added into the ROI.

There are many proposed ways to estimate or speculate ROI. Studies may suggest methods for your company’s project based on size or scope, but these proposals will never be as good as testing the process in your environment and measuring your own results. Scrum and other Agile processes have great values and intentions, stressing business value and expediently delivering a better product, but that still does not mean that these are the best practices for you, your company or your project. Furthermore, you must take into consideration that when learning a new methodology, a team’s initial attempt may not be representative of the best work that can be output with that methodology.  It is important to find a method that fits you and your team.