For the past decade, insurers have been investing in digital transformation to bring innovative new products to market and changing the way that product engineering is practiced to keep up with the ever-changing needs of their customers and maintain their position in a highly competitive marketplace. To this end, many have now undergone agile transformation, including the implementation of automated testing within CICD pipelines. The insurance industry is therefore seeing a growing trend with more organizations needing help, guidance and solutions related to Test Data Management (TDM).
This process change is one of the key drivers for TDM initiatives, as many organizations require highly consistent and representative test data across their heterogenous technology landscape to enable accurate testing within CICD.
Many insurance organizations have implemented workarounds to create data, but many of these utilize sluggish data refresh processes which can also introduce data security risks. For example, taking copies of data, which can contain millions of records, and pushing it into the lower environments can slow down the SDLC and is incompatible with CICD. In addition, legislation such as GDPR and Data Protection Regulations mean that organizations must use, store and access data responsibly, avoiding the use of live customer data for testing purposes, which could result in data leakage, misuse, and potential legal action.
The test scenarios within insurance can be complex, with data required in specific states to trigger specific underwriting and pricing rules. There are many time/date driven test scenarios, requiring policies of a certain age to trigger processes such as renewal, policy lapse or regular communication from the insurer. In addition, the solution architecture may require data to correlate in several databases and systems. The creation of this data can be labour intensive, time consuming, costly, prone to error and can hinder the progress of time savings introduced by automation and CICD initiatives. Therefore, insurers require a way to quickly synthesize accurate and representative test data.
TDM best practices require data to be de-identified before being used for testing, using techniques such as data masking, where words or characters are scrambled, shuffled, or replaced to obfuscate real information. Other techniques include the use of pseudonyms or swapping addresses and dates. Although most organizations are now turning to Synthetic Data Generation as a method to create reliable test data and ensuring there is no privacy or regulatory risk.
Automated Synthetic Data Generation Tools can quickly create data for all test requirements, by simply configuring the attributes, combinations, and logic within the tool. Data can be generated with different ages, genders, addresses, vehicle registrations to create the required permutations and appropriate volumes. The data can be synchronized across multiple applications and databases to ensure correlation of data for end-to-end test scenarios. The generation is orchestrated as part of CICD pipelines, and the dependency on database administrators to refresh the data is removed, improving speed and efficiency.
More recently, Generative AI has been utilized to complement synthetic data generation, with models trained on organization specific data, used to produce close to production like data for testing. Here at Qualitest we have incorporated Gen-AI capability within our accelerator to produce test data for testing microservices.
Another method that can be used is Data Virtualization, which facilitates the virtualization of data assets derived from production or other environments. With this approach it is necessary to also use data obfuscation techniques to ensure the data is anonymized and compliant. There are enterprise scale solutions like Delphix and IBM VDP/Actifio which can be used to virtualize various types of databases and are compatible with CICD and DevOps processes.
In most cases there is no effective standalone solution for TDM, and a combination of solutions are required to meet all requirements. It is important to thoroughly understand the problem statement and requirements to ensure the appropriate TDM solution is defined.
Whilst there is an abundance of TDM tools now available to choose from, it is important to call out that simply purchasing and implementing a tool in isolation, is unlikely to yield the desired results. Tooling is one part of the jigsaw, but like any technical solution requires the application of other Engineering practices such as Solution Architecture, Solution Engineering, the meticulous gathering of data requirements, testing and ongoing maintenance. As the system under test evolves, so do the requirements, tests, and underlying data requirements, thus resulting in TDM becoming a capability required within an engineering function for an ongoing basis, including the right people, processes and technology.
Effective test data management is indispensable for insurance companies striving to maintain their competitive edge in an ever-evolving industry. By prioritizing data quality, security, and compliance, insurers can reduce risks, improve the efficiency of their testing processes, and ultimately deliver more reliable and customer-centric solutions. As insurance continues to adapt to new technologies and regulatory changes, robust test data management will be a key factor in ensuring the industry’s resilience and success.
Qualitest has a Centre of Excellence focused on TDM, with experienced consultants who can support our clients with custom TDM solutions, designed to help them achieve their transformation objectives and the highest levels of data privacy protection. We also have partnerships with several synthetic generation providers such as GenRocket and Synthesized.
