Insights Blog Qualitest Helps Telecoms with Customer Experience Analysis

Case Study

Qualitest Helps Telecoms with Customer Experience Analysis

This project focused on in-house abilities to learn and extract customer behavior data through COTS data analysis tools, instead of deploying new website pages or functions. With no devoted development team, scrum master/test lead or PO, the outcome fell mostly on one team’s DDM (digital development manager) and a QA. Targeting the overall site required a good understanding and familiarity with many different site sections and functions.

Client Overview

The client is a telecommunications and internet service provider, providing 3G and 4G services through its own network infrastructure.

Business Needs and Objectives 

To help optimize their website, the client sought tools to enhance insight analysis into consumer behaviors, likes and dislikes of the site’s functionality and usability. The customer experience analytics tools (crucial for building good retail CRM) chosen were ForeSee and Decibel, which track, record and measure user behavior and satisfaction. Both tools had been previously utilized on the site, collecting session data and customer input. This project involved deploying a new ‘Replay’ function that both tools had but that the client had never used, training the client on proper usage, and ensuring expected behavior and privacy.

The ‘Replay’ function allows analysts to watch customer’s online sessions after the fact. In addition to ensuring that the replay function was collecting and showing data correctly across browser and mobile device fragmentation, the client’s top priority was ensuring that this function complied with legal privacy requirements specifying that all customer personal identifiable information (PII) must be obscured and hidden during replay, including PII entered in forms by online users and PII saved previously on the site in personal accounts.

The Qualitest Solution:

We integrated a higher level of quality QA into the website development scrum teams. Qualitest consultants filled the gap with experienced, fast thinking, fast learning and fast adapting testers in those teams. A Qualitest Test Consultant was integrated into the scrum team in charge of the ongoing site wide optimization efforts of the client’s online channel.  The 3-month effort was driven by the team’s DDM (digital development manager) and myself as the team’s QA. Instructions came straight from the stake holders while the two of us worked on it with the insight team.  We concentrated on:

1. Studying the ForeSee and Decibel tools:

  • Learning how each tool collects data from the site and how we can manipulate the collection
  • Learning how to use each tool as a front end user
  • Prompting the different tool functions deliberately for the testing process

2. Reviewing all PII on the client site:

  • We received an initial list of site pages to review and that include PII from a UX
  • All these pages had to be tested with both the ForeSee and Decibel Replay functions

3. Working with 3rd party developers to fix any defects found (PII’s that weren’t obscured correctly)

4. Documenting procedures to follow by all development teams when introducing future new pages or functions that include PII

5. Updating the site’s cookie policy to include information about the new functions

Separate test script templates were constructed for Decibel and ForeSee, based on the specific build and functionality of each one. Two sets of scripts were compiled and run using each template on every single page and function that showed PII on the client website.

All scripts were run on the top browser and top mobile phone used by client customers at the time. In addition, only Decibel scripts were run on a tablet since ForeSee was not configured to record tablet sessions. After testing these main browsers, a smaller sample of scripts was run across additional browsers and mobile devices according to a test matrix agreed upon with the development manager and the insight team.

Defects and fixes were managed in two avenues.  While changes to the recoding procedure in Decibel could be managed by a client analyst, ForeSee changes needed to be sent to the 3rd party developers in another country, making the process take longer and become harder to complete.

All test templates were later entered to the test scripts repository for future use by the QA team.

Key Benefits

Key benefits resulting from the testing solution for this project included:

  • Reusable test scripts and project delivery procedures when delivering a function with PII
  • Training for all of the QA team on both insight tools created run by a Qualitest Test Consultant
  • Full cross browser and devices coverage according to the agreed upon test matrix
  • Complying with legal requirements
  • Full satisfaction from key stakeholders