Results Based Testing (RBT) is a new software testing pricing model that sets forth the expected value to be delivered by the Testing Teams.
Imagine your testing provider making a commitment to find 97% of the defects in the system. Can that be measured? What would that be worth? What would be the impact on the customer-provider relationships and accountability? How can you bind that commitment? This white paper will attempt to answer those questions.
Very often the question “what budget should I allocate for testing” is asked. The answer to this question is somewhere between zero and infinity. There are many elements that define the budget for testing, such as complexity of the system under test, amount of iterations of testing and of software releases, quality of development and of unit testing, efforts spent by development on present release and past releases and more.
However, an important factor for defining the testing budget is the level of risk the organization is willing to take. No risk means we need to produce a bug-free system (and therefore the testing budget will be close to infinity). You may spend one day to test a system and if you’re a good tester, you will probably find lots of bugs or you could spend 3 months with a group of 50 testers to test the same system. The difference would be with the amount of defects you find and more important – the amount of defects you DON’T find…
Results Based Testing usually involves three elements:
RBT is normally utilized when part or all of the software testing process is outsourced to a third party and a core contractual SLA together with a pricing mechanism sets the exact payment made at each SLA level. The pricing mechanism may be a flexible rate for each SLA level or a Penalty/Reward mechanism, all with the goal of creating an incentive for the Testing Supplier to meet the business targets (results) set forth. However, RBT may (and should) also be used for internal testing teams even though in such cases a penalty/reward mechanism is harder to implement.
Another good use of RBT is for establishing the necessary framework for measured continuous improvement in which results from previous periods can serve as a baseline for following period’s targets.
When evaluating the level of testing, several Key Process Indicators (KPIs) should be measured. However, in order to keep things simple, the main focus should be on two main questions:
Although it is clearly important to be able to answer the above questions, most organizations fail to measure these two KPIs and thus, are unable to provide accurate visibility into the quality and efficiency of testing. Below we will present a simple method for measuring test coverage which will enable us to answer the first question and a method for setting a budget for achieving these results, thus answering the second question.
There are many other indicators such as staff level expertise, attrition level, reporting to stakeholders, communication with development teams, delays in the process and more. However, all other aspects feed directly into the two main questions above. For example, if the level of staff is not sufficiently high, the percentage of defects found will decrease and/or the cost of testing will increase.
In order to measure the percentage of defects found by testing (a type of test Coverage KPI vs. the escaped defects KPI) the organization should utilize the following process:
Only defects in the last status (New Defect) are counted for this metric.
The process above, although not being used by most organizations, is extremely important in case the organization is looking to start implementing RBT and to continually improve the efficiency and effectiveness of the testing process.
Measuring the test coverage is done by dividing the amount of defects found by the testing provider with the amount of defects found by the users of the system.
Since critical defects carry a different importance to the organization versus less severe defects, each defect is multiplied by its severity. For example, assuming a scale of 1-5 is used, a critical defect (severity = 5) will be counted in the same way as 5 minor defects (severity = 1).
Only defects found in a certain period after release of the system shall be counted (normally this is defined as 3 – 6 months).
The Testing Budget Calculator is a formula used to define the testing budget required for testing a system or a version of a system.
There is no one generic formula for calculating the testing budget for all types of systems since the total effort required depends on the organization and the system specific characteristics.
First we need to identify the parameters that affect a testing budget. These factors might include:
When selecting the relevant parameters, the organization should verify it has access to the values of each parameter as any change in the value has implications on the testing effort required. It’s nice to have sophisticated parameters like lines of code, but there’s no use in using such parameters if there’s no access to the actual values added in each version or in case the amount of lines does not change the total effort required for testing.
The next step is creating a formula using all relevant parameters. It’s important that the formula is as simple as possible allowing for certain assumptions based on past experience. An example of a simple calculator might be a set percentage of the development budget. Please see engagement examples in the examples section below.
In traditional software testing outsourcing engagements, the testing provider provides the service and the customer pays for the services, either at a fixed price, on a time-and-materials basis or a cost-plus model.
Results Based testing is the only gain-sharing approach in which long term interests are perfectly aligned. This model encourages collaboration and creative problem-solving as both parties work toward common business goals. It also provides the supplier a greater freedom to determine how to better achieve the results.
The basic pricing model is Managed Testing Services based on a pre-defined rate card with only a certain amount paid (usually 70% of the total amount) while the final calculation of the cost is made once actual results are calculated.
According to the difference between the actual hours multiplied by the rates and the expected budget that had been calculated in the Testing Budget Calculator, the following adjustments can be made:
The following adjustments are made based on the actual test coverage KPI:
For example: assuming the coverage KPI target for all defects is 95%, the adjustment mechanism will be as following:
| Actual Coverage | Penalty/Reward |
| 0%-80% | 30% penalty |
| 80%-86% | 20% penalty |
| 86%-92% | 10% penalty |
| 92%-98% | No penalty/reward |
| 98%+ | 10% reward |
Additional Adjustment can be defined based on other agreed and measured KPIs
| Large Bank | Tier 1 Defense Contractor | Telecom Equipment Manufacturer | ||||
| Size of testing team | 120 testers | 160 testers | 140 testers | |||
| Main SLA | Coverage | 96% | Coverage – all defects | 97% | Coverage | 85% |
| Average delays in schedule | 5% | Coverage – critical defects | 99% | Average time to recruit new staff | 21 days | |
| System up time | 99% | Satisfaction surveys | 3.8/5 | Attrition of key employees | 5% | |
| Location | Qualitest Test Factory | Qualitest Test Factory | Qualitest Test Factory | |||
| # of parallel testing projects | 60-80 | 20-30 | 8-10 | |||
| Average duration of each testing project | 15-30 days | 1-2 years | 3-4 months | |||
| Testing Budget Calculator | Set % of development budget | Formula with 6 factors: Development man days, # of iterations, efforts in previous versions, ATP required and more | % of development man days | |||
| Pricing model | Fixed Prices based on the testing calculator | T&M. If the actual cost is above Testing Budget, a 30% discounted rates applies If the actual cost is below budget – 20% reward | T&M. If the actual cost is above Testing Budget, a 30% discounted rates applies | |||
Results Based Testing enables organizations to:
Provides the customer flexibility to scale the testing up or down in line with its business needs.
