The Client is one of the largest American non-profit news agencies, who has been leading press coverage and used by numerous news and media outlets around the world for over a century.

The Client employs more than 3,000 people globally and is known as the news source of the world’s leading web portals, who rely on the Client for coverage of breaking news.

The growing demand for news, especially during a period of extreme global changes, requires the Client’s to be nimble and adaptable, deliver instant – reliable – frictionless value on scale and offer a better user experience for the countless people who rely on their news cycle.

Millions of users demand and deserve the best experience

Due to the dynamic nature of their business, the Client’s testing team had difficulty supporting the changing product needs arising from market fluctuations. Lack of specialist testing skills and inadequate capacity in complex multi-vendor development and globally-distributed teams resulted in delayed releases, production quality issues and prolonged testing cycles with escalating costs, putting projects at risk and impacting revenue.

A number of big news events had recently shown their infrastructure was unable to handle peak loads, degrading the customer experience with degradation to response times and intermittent outages.

The Client’s existing automation framework did not support the rapid pace of development in an Agile setup. This led to a huge backlog of eligible automation test cases in the regression suite over multiple releases, resulting in longer execution times, and in-sprint automation had never been realized. The automation framework was not generic so it could not be re-used across different applications, resulting in lower re-usable components and exhaustive maintenance efforts and enhancements.

It also lacked integration functionalities to connect with 3rd party tools, like Jenkins and MS Teams, to achieve continuous testing and get instant reporting.

“The existing client automation framework did not account for the rapid pace of development in an Agile setup.“

The Client wanted:

  • A Quality Engineering partner who understands its industry and business challenges, with the capabilities and thought leadership to drive business value and enhance the customer experience by improving quality and facilitating transformation across the development lifecycle.
  • To implement a “shift-left” approach to its Agile development cycle by implementing in-sprint and continuous testing, for both functional and non-functional aspects assuring User Experience by Design.
  • A universal testing solution based on industry best practices and incorporating automation for functional and non-functional testing across its 60+ internally developed applications to reduce testing cycle times, increase cross-platform test coverage and decrease costs while focusing on quality and user outcomes.

Optimizing automation

Qualitest deployed senior transformation architects to build the roadmap to the Client’s target dates and designed business outcomes.

Beginning with functional testing activities, we reviewed the existing regression test suites, optimizing where applicable and creating additional test cases mapped to the user stories/features using exploratory testing techniques, with quick adaptation to the Client’s agile development process.

After quick dry-run executions and functional conformance, they were categorized into Sanity and Regression by relevancy and then handed over to the automation team to develop the automation scripts.

“To progress automation, Qualitest started with the applications using Qualiframe, our proprietary hybrid framework, which involved GUI, backend, and API.”

Test Impact/Root Cause Analysis was performed for each change introduced using Risk Based Testing Techniques and relevant test cases were executed/updated to validate the functional conformance.

This was the continuous process followed to build the automation test bed and became very effective during hotfix or rollback testing; this paved the way for in-sprint automation and uninterrupted automation execution using CI/CD tools.

To progress automation, Qualitest started with the applications using Qualiframe, our in-house developed hybrid framework which involved GUI, backend, and API. The framework follows best practice design architecture and element locator strategies to handle the application changes for each release and reduce the maintenance of automation scripts drastically. The results from this impressed the Client, who gave us the green light to further expand continuous testing and automation activities across multiple platforms to help with the overall user experience.

The Automation Framework was configured to address the crucial data validation challenges presented by the Client’s applications and business processes, by developing utilities. This increased the efficiency of data comparison tests between legacy and new data providers which involved testing of thousands of records. This also ensured comprehensive volume coverage within a very limited timeframe.

To perform distributed and remote test execution across multiple browsers, Selenium Grid was implemented, which contributed to increased test coverage in reduced time.

Shifting left

To reduce the production slippages that the Client had been experiencing, Qualitest proposed a Shift-Left approach to their agile development life cycles. By aligning with the development teams in a close-collaborative model, whether the Client’s inhouse resources or vendor development teams, QA was able to get involved in testing much earlier – utilizing Jenkins to execute testing earlier.

Integrated automated tests into the CI pipelines are triggered at a regular cadence and whenever a build is deployed, the test suites are executed, and notifications sent out to all the stakeholders about the execution results/stats. This helps identify the application issues much earlier in the software development cycle, informing development, performance engineering and engineering quality into the product from the outset.

The resulting quality increases allowed for accelerated delivery as well as decreased production incidents.

Engineering performance

Based on recent issues, the Client knew they had an immediate challenge in the infrastructure performance and needed rapid resolution to ensure they were able to support the level of users they were experiencing during high-profile news events.

Qualitest needed to assure peaks of 5,000,000 users within 5 minutes could be handled without degradation or failure on the new platform architecture on AWS.

Qualitest undertook enterprise-level deep dives and fact-finding excursions into the Client’s existing production systems. Qualitest gained a detailed understanding of frequently used business processes, volumetric information, system usage, SLA, and others. At the end of this phase, Qualitest provided a precise performance testing strategy, along with tools that would be used for the steady state project support. After the deep dive into several proof of concepts, Qualitest identified the right solution to run these mega performance tests. Building on the trust developed over many years partnering with the news agency, Qualitest recommended the use of Apica as the performance testing platform.

Tools identified:

• Google Analytics (for production tracking analysis)

• Apica Systems

• New Relic / CloudWatch / Kibana / Akamai (in Production)

The root cause of the problems the Client has experienced was identified as autoscaling, and several critical risks which could have seriously impacted performance were identified and addressed. Autoscaling was turned off and the many incremental tests helped determine the minimum and maximum number of servers required for “mega” events.

Following the remedial action to assure the platform could handle well in excess of previously seen user levels, Qualitest implemented continuous monitoring solutions into the live environment, and embedded automated performance testing into the CI/CD pipeline to ensure future developments did not impact performance.

Key benefits

Qualitest delivered an Agile Transformation, cost savings and quality improvements to the Client’s timelines:

Achieving business agility with a faster time to market:

  • Qualitest helped the Client bring down the regression cycles from months to days and in some cases – hours.
  • Qualitest automated over 95% of test cases by the end of the release cycle, greatly reducing risk, increasing quality, and speeding up delivery.
  • The client increased their release throughput by 64% overall through automation and by integrating automated tests into CI/CD pipelines, enabling them to put updates live faster, enhancing the user experience and engagement.

Cost savings:

  • With continuous automation integrated into the CI/CD pipeline and a reusable low-maintenance framework, Qualitest has helped the Client save upwards of $500k in development, defect fixes and testing spend in the last two years.
  • Expedited test automation implementation was provided as a fixed price service, resulting in lower costs and assurance of outcomes for our Client.

Quality and user experience improvements

  • Built-in quality: A high level of automation coverage helped execute the automated tests at a higher velocity and much earlier in the development cycle causing less production misses and reducing the number of hotfixes and rollbacks from a total of 20+ to two over a period of five years.
  • Integrated performance engineering ensures the reliability and scalability of the platform so the Client could concentrate on reporting the news, and not become a news item themselves.

“The client increased their release throughput by 64% overall through automation and by integrating automated tests into CI/CD pipelines.“

quality engineering free assessment Download the PDF