Qualitest Implements Test Automation Factory
Qualitest drives test process improvement and dramatically increases test automation coverage to implement an Agile environment.
The Client, the healthcare industry’s most comprehensive provider of cost management solutions, acts as an intermediary between its nearly 900,000 participating healthcare providers, and its 1,400 clients providing claims re-pricing services at an estimated volume of over 40 million medical claims per year.
Business Needs and Objectives
The Client selected QualiTest to help drive test process improvement and dramatically increase test automation coverage following a lengthy RFP process, which evaluated several large testing and IT service providers.
The goals of the Client included the following:
- Establishing a QA partner who was reliable, trustworthy and with extensive expertise in the software testing industry
- Achieving an optimal level of automation for their 150+ internally developed applications
- Creating a testing solution that would decrease prices while increasing test coverage with the best tools and resources for the job
- Establishing a “Test Automation Factory” environment to rapidly close the deficit of disparity between manual testing and automated testing, and ramp up the automated test efforts
- Developing testing frameworks for Electronic Data Interchange (EDI), Web Services, Extract Transform Load (ETL) and regression.
- Develop a flexible solution, customized to the client’s needs and with little technical expertise required to run the delivered test suites.
The first few weeks of engagement between Qualitest and the client were involved with tool selection. After extensive tool comparisons, Qualitest and the client decided to proceed with using Selenium, based on the aspects and requirements of the project.
The Qualitest Solution
Leveraging Qualitest’s right-shore testing model, test leadership resources are located onsite at the client’s Illinois and New York City offices, and supported by a test team of onshore test leads at the QualiTest Connecticut Center of Excellence and another test team at the Qualitest India site.
QualiTest and the client decided on a 70/30 model when strategizing the Right-Shore testing model, with 70% of resources located offshore and 30% of resources located onshore. Although this model is a higher percentage than the usual recommendation, the allotment was necessary in order to provide for a smooth knowledge transfer. The knowledge transfer was more complex than in most cases because all domain knowledge from the client was located onshore and largely undocumented.
Qualitest’s comprehensive consulting service aligned with the client’s Software Development Life Cycle to identify their key business processes and requirements, understand their business and infrastructure and assist in configuring a performance testing environment that accurately simulates production environment. The testing included front-end and back-end testing for web-based applications that were developed in-house by the client.
Some of the key functions performed by the Qualitest engineers include:
- Identification and development of an in-depth understanding about the client’s applications and system architecture for the purpose of application test automation and improved regression coverage through Selenium.
- Development of a scalable, maintainable and user-friendly Selenium automation framework leveraging the Keyword Driven Testing and Data Driven Testing methodologies. This framework was especially designed to allow testing professionals other than automation engineers to be involved in producing, executing, and analyzing the automation suite and test results.
- Reduction in the turnaround time of regression suite execution across dozens of applications; in many cases this reduced the execution time from weeks to days or hours.
- Creation of scripts and scenarios to produce consistent, measurable and repeatable regression tests. These tests and scenarios are designed to mirror the client’s live production environment, business patterns and throughput, including projections for future growth when applicable.
- Improvement of regression test coverage by providing specialized personnel for the development of additional test cases for new and existing test suites. These test suites were designed with an aim toward Selenium test automation.
Challenges Discovered and Addressed
Managing Claim Data
Due to the challenge of managing consistent and reliable claim data for consumption by many applications, the client needed a solution that allowed them tight control over claims in the database without creating a dependency on an application that needed testing to begin with. QualiTest created a configurable solution for loading baseline data directly to database tables. By changing some simple values in excel workbooks, this process was able to manage and load claim data while applying important pre-processing and data scrubbing.
Knowledge Transfer and Test Design are Complex
Most organizations possess strong and valid concerns for how an outside consultant or contractor in any setting can effectively learn applications, business rules, and industry knowledge quickly and effectively enough to design and implement a successful automation suite. In order to solve this challenge, dedicated test designers were assigned to individual groups which worked very closely with client subject matter experts in order to proactively and cooperatively develop appropriate automation strategies, designs and documentation.
Writing Test Cases for Automation
Test cases written without automation in mind can raise some interesting challenges. In many settings, the best mechanism for knowledge transfer and assessing automation coverage is often represented by how an organization did or would test the application manually. When it came time to write additional test cases for more automation to be developed upon, the client lacked the experience in designing these tests for the purpose of automation. Qualitest allocated additional resources that worked closely with client personnel to gather testing requirements and produce test cases which could be effectively executed manually and properly transitioned to an automated test suite.
Handling EDI File Formats
Much of the work involved with processing medical claims either consumes or produces EDI files. These files come in several different formats, all of which have their own complexities. The format specification for X12 alone can be hundreds of pages in length. The client desired an automated solution which could identify errors in these files with a high rate of accuracy, but the options of developing complex parsers or acquiring commercial tools came at a prohibitive cost. QualiTest developed a heuristic for parsing such files into meaningful components without creating an unnecessarily complex solution. This enabled the client to spend less time manually verifying file contents, integrate file comparison tightly into the automation, and maintain a high level of confidence in the results of the automated tests. This portion of the framework was designed with growth in mind and can be easily expanded to support additional file formats.
Some benefits that have resulted from Qualitest’s ongoing managed testing services include:
- Saving hundreds of thousands of dollars by using an opensource tool
- Gaps and risks are identified early in the process, reducing bugs at later stages and avoiding project delays.
- Advanced tools, metrics and techniques allow stakeholders to track the statuses of projects
- The automated testing for the in-house developed applications reached over 65% coverage within the first year, reducing the execution time from weeks to days or hours.
- The client was able to implement a “shift-left” approach to their Agile development cycle. Because the testing was implemented and completed within an earlier time frame, development and QA departments were able to interact from an earlier stage and applications were entered into “go-live” production in a more expedient manner.
- The client received expedited test automation implementation at a fixed price service, resulting in lower prices.