Insights Blog Proving QA’s Power to Protect a Tech Giant’s Brand Reputation

Case Study

Proving QA’s Power to Protect a Tech Giant’s Brand Reputation

The Client's overall QA function has evolved and has started to become standardized across their organization. Qualitest enabled test coverage to increase by up to 30% and defect leakage to production to decrease by almost 60%.

Overview
Challenges

QA had been conducted by Client engineering teams and developers.

Leaked defects and poor product quality put brand reputation at risk.

Solutions

A phased, collaborative approach built willingness to incorporate QA teams.

Audits and performance metrics showed clear benefits of specialized QA.

Results

Client increased QA-supported engineering teams from 19% to 47% within 18 months.

Defects leaked to production decreased by 60% with QA involvement.

Client overview

The Client is a major technology giant who leads the industry in multiple sectors. The Client has more than 2 billion active daily users of their products, which include a payment platform/digital wallet, an AR-VR product, an eCommerce platform embedded with an AR-VR products catalogue supporting hyper-personalization, an eLearning platform, a Chat platform and many other similar industry-leading technologies.

With its extensive range of products and a massive, demographically diverse user base spanning all geographies, our Client needs to stay ahead of the curve, ensuring quality and constantly innovating and growing to meet customer demands.

Rescuing a reputation from the dangers of “dogfooding”

In the Client’s product-centered organizational structure, most product engineering teams perform their own QA, along with developers–a practice called “dogfooding”–with no involvement from a specialized QA team. This process had been in effect for the Client’s launch of multiple new features, with disappointing results.

A lot of releases needed rolling back, and some features introduced to a limited geographical sample performed too poorly for exposure to a wider market. Multiple nonfunctional defects were leaked to production, attracting bad press.

Poor product quality was putting the Client’s brand reputation at risk, and they asked Qualitest to step in. They wanted us to develop and execute a tailored QA approach to build confidence among engineering teams and senior stakeholders that dedicated QA teams could make a difference.

“The project was a success and enabled Qualitest to expand the presence of QA within the Client product teams.”

The Client’s specific objectives were to:

  • Establish a QA partner who understood its industry and business rationale, with the eventual goal of a multi-year engagement.
  • Understand the current work environment and organically induce change.
  • Increase the QA presence among the product engineering teams.
  • Prepare a tailored approach for each of its diverse products and introduce standardization in terms of templates and common goals.

Building trust in QA with an iterative, data-backed approach

We faced some challenges from the start:

  • The Client’s well-known AR-VR product has more than 400,000 creators from 190+ countries, generating 1Billion+ views. Augmented/Virtual Reality technology is emerging, so finding case studies and a template appropriate for the product was not going to be easy. Qualitest was commissioned to take charge of overall product quality.
  • Our Client’s payment platform/digital wallet is a game changer in the remittance industry. The Client asked us to support QA in the product’s compliance features, a high-risk area that is of paramount importance for brand value in the market.

“Overall QA function has evolved and has started to become standardized across the Client’s organization.”

Our teams designed and implemented an iterative process model with three clear, well-defined phases.

Phase 1: Identify Early Adopters

In this phase, Qualitest QA engineers, backed by QA managers from the Client side, spoke to teams who were willing to accept outside quality professionals into their group. Client leads agreed on the areas of support and set goals for strategic initiatives to improve product quality.

Qualitest identified the following deliverables for the first group, the Early Adopters:

  • Support new feature release.
  • Support production defects & fixes.
  • Support QA manager with strategic initiatives.
  • Report to middle and senior leadership on BAU and strategic initiatives progress.
  • Collate data/wins/lessons learned to share impact with wider teams.

Phase 2: Data-Backed Pitch

Applying several QA metrics, Qualitest engineers demonstrated the benefits Early Adopters had achieved from our QA team’s engagement. The project expanded with a second group of teams to increase the support level, with similar deliverables identified.

  • Support new feature release.
  • Support production defects and fixes.
  • Support QA manager with strategic decisions.
  • Report to middle and senior leadership on BAU and strategic initiatives progress.

Phase 3: QA Audits

Finally, our analysts started performing QA audits on deliverables for the Client teams who did not or could not engage our QA teams. As part of these audits, we started observing the following:

  • Processes around scripting.
  • Automation coverage.
  • Regression coverage.
  • Automation script stability.
  • Defect density.
  • Defect leakage.
  • On-time delivery.

Key benefits

The project was a success and enabled Qualitest to expand the presence of QA within the Client product teams. Our QA team’s involvement, which started with one product, currently supports 14 products.

In addition:

  1. Overall QA function has evolved and has started to become standardized across the Client’s organization.
  2. Process standardization saved several hours of QA’s time in agreeing on the documentation templates.
  3. Test coverage has increased by up to 30% for the products covered.
  4. Defect leakage to production has decreased by almost 60%.
  5. Improved process on scripting resulted in optimized scripting standards, reducing the script maintenance time by 38% for applications under test.
  6. Automation script stability improvements resulted in reducing flakiness to almost 80%, thus saving a crucial average 17 hours per release from go-live timeline—of paramount importance to our Client.
download the pdf