Public Sector Institution Implements DevPerfOps Framework to Accelerate Digital Transformation

project highlights

01 — Challenges

The Client had struggled to adopt a “Shift-left” approach to performance testing due to the absence of a standardized automated framework.
Problems included asset duplication, unchecked security vulnerabilities, high manual testing efforts, and a lack of real-time monitoring and feedback.

02 — Solutions

An extensive performance test framework was implemented to overcome these challenges and improve efficiency.
Promoted team collaboration via source control, streamlined cloud-based test deployment, and implemented automated testing through CI/CD integration.

03 — Results

A centralized repository was established to store all reusable assets, ensuring strict security compliance with no critical or high vulnerabilities.
Automated cloud-based performance testing reduced data and report generation time from approximately three hours to 30 mins, while real-time monitoring provided immediate feedback to teams.

Client overview

The Client, a public sector entity, serves as a vital link connecting people and businesses by facilitating the exchange of letters, parcels, and goods both domestically and internationally. With a network of 11,000+ branches across the UK, they ensure seamless and timely delivery.

Insufficiency and gaps in performance testing

They wanted to transition from their faulty colocation (CoLoc) system to AWS Cloud to achieve a modern IT architecture and leverage the benefits of cloud computing. Despite adopting DevOps and collaborating with multiple vendors, they lacked a well-defined structure to expedite performance testing. Some of the issues they faced included:

Single team dependency: Performance testing was handled mainly by one team.

Argo CD to GitHub Actions transition: Required a shift in their deployment pipeline.

Lack of squad-level non-functional requirements (NFRs): Impacted early-stage integration of performance testing.

Inefficiencies and manual efforts:

Two hours of daily manual effort for data generation.
Frequent Dockerfile revisions leading to large images.
Repeated processes due to daily AWS Token expiration.
Manual effort to set up AWS jobs.

Lack of real-time monitoring and feedback:

No immediate feedback until report generation.
Manual report creation process.

Automated performance framework: from proof of concept (POC) to cloud-based efficiency

Qualitest leveraged a proof of concept to develop and evaluate two potential solutions, focusing on their strengths and weaknesses. This led to the creation of a framework designed to prioritize automation and reduce reliance on operations teams.

Standardized automation performance framework: The framework featured a centralized repository, where all reusable assets were stored. This setup ensured compliance with client security policies and effectively eliminated critical and high vulnerabilities. Additionally, cloud-based automation was implemented to streamline the setup of load test executions, integrating with the CI/CD process to support both on-demand and continuous integration testing.

Efficiency enhancements: Automated data generation eliminated the daily 3-hour manual effort previously required for data creation. Docker processes were optimized by reducing the frequency of Dockerfile revisions and minimizing image sizes. Additionally, the setup of AWS jobs for performance tests was automated, which significantly reduced the need for manual interventions.

Improved security and monitoring: Security checks were embedded within the performance testing framework, allowing for proactive identification and resolution of vulnerabilities. Real-time monitoring tools from Datadog were implemented to provide immediate feedback on test results. Additionally, the report generation process was automated, delivering instant feedback on the success or failure of performance test jobs.

Comprehensive testing and framework adoption

Targeted testing:A comprehensive test pack was developed, emphasizing goal-oriented and SLA-based testing driven by Key Performance Indicators (KPIs). This approach ensured that performance testing was closely aligned with specific objectives and service level agreements.

Managed execution:Test execution was outsourced to a fully managed AWS Batch environment, with oversight and control provided through GitHub Actions. This arrangement streamlined the testing process and reduced the need for direct intervention.

Educational technical sessions:Educational sessions were conducted with squad members to foster early adoption of the new performance framework. These sessions aimed to identify potential performance issues early in the development lifecycle and ensure that the team members were well-equipped to utilize the framework effectively.

Key benefits

Integrated a configurable framework into the CI/CD pipeline, enabling continuous performance testing.
Completely automated data generation and report preparation, saving significant time and effort (~2.5 hours).
Enforced development and security best practices, resulting in zero critical and high vulnerabilities in testing assets.
Removed the need for manual configuration of performance test jobs in the cloud.
Ensured efficiency and consistency across various testing environments.