Patch Management Testing

Qualitest has led multiple enterprise level Regression testing efforts as well as test automation efforts integrated into enterprise patch management efforts at midsize and large global organizations. This experience has resulted in the following comprehensive methodology and set of best practices.

General

Software complexity, layering and integration continues to increase and with it the challenge of testing. While testing new systems or new features in existing systems has become integral to the system development lifecycle, software is still and probably will never be completely free of bugs. The industry as a whole has thus accepted an approach of post release fixing also known as patching.

Patching can take many forms and may relate to many software layers such as patching of operating systems on clients (workstation, mobile…) and servers, to patching of infrastructural software and firmware components such as web or application servers, JVM and .NET versions, network equipment firmware, security software and basically anything that runs software. As each of these layers is developed and tested by a different vendor and the number of combination and configurations is infinite these vendor’s testing, as rigorous as it may be will never be able to identify all issues in the lab. This means the responsibility for finding such integration and configuration problems falls on the organizations that use the software. This in turn results in the need for comprehensive testing that can be executed as often as needed and at a reasonable cost.

Testing activities and testing scope

Patch test design requires a different approach and scope then traditional functional testing. The differences mainly come from the fact that patches defaults typically break applications completely or not at all. When applications break they normally either fail immediately when run or in areas of integration to other applications. A comprehensive patch test process should focus on the following 2 areas:

• A system level, end to end sanity test across all main business processes in the organization
• Complete coverage of software integrations both internal to each system as well as across systems.

Test Activities should include:

• Generate checklists of applications, integrations and end to end business processes.
• Test to validate that patches do not cause conflicts with coexisting applications.
• One or more test suites should be developed that exercise the functionality of the system and the test suits should be kept in a library or tool.
• Tests should be conducted, and procedures written, to verify that the installed patch can be removed without impacting operations in case problems are identified post role out.
• Checklists and procedures should be used for patching activities to ensure both initial accuracy and repeatability of patching activities and testing.
• Records of the patch, tests, and configuration changes should be logged and documented in the configuration management record.
• Specific tests should be planned to verify that the patches applied fixes the intended problem/s identified by the supporting organization.

Test Environment Considerations

• Establish a dedicated testing environment for patch testing.
• The testing environment should simulate the production environment as closely as possible.
• Conduct a gap analysis between the production environment and available test environment.
• Conduct a risk management process (identification and mitigation) for each gap identified above.

Manual Vs. Automated

A decision on which patch tests should be manual and which patch tests should be automated should be based on similar considerations to these followed with any testing activity, the main factors to be considered are:

• Can these tests be automated?
• Will the systems considered for automation remain in operations for a sufficiently long duration?
• Are there any benefits to automating tests that go beyond efficiency/cost savings?
• What efficiencies can be achieved by automating? To answer factors such as the following should be considered:
  • How often will these tests be run?
  • How long and how many resources are required for manual execution?
  • How long and how many resources are required for test automation? For test creation?
  • What would be the additional hard costs required for automation such as test tools or additional test environments?
  • What are the expected costs for maintaining these automated tests?
• A comprehensive test automation approach is described in a separate included set of articles
  • Test-Automation-From-the-Ground-Up:  www.qualitestgroup.com/Test-Automation-From-the-Ground-Up

Typical steps in creating a patch testing program