Qualitest Helps Leading American Broadcasting Company Automate Reporting, Drive Strategic Business Plan
Qualitest created automated real-time dashboards to cut reporting time to practically zero.
Standardize and simplify reports on QA activities across varied teams and projects, to better assess automation and testing ROI.
The team was wasting precious time on generating reports manually and needed better tracking and reporting capabilities.
Qualitest leveraged its analytics product, Qualiview, to build real-time dashboards.
We gathered data from 4 systems – Jira, ALM, Jenkins and Bitbucket – to show consolidated reports across teams and individual projects, in real-time.
Automated real-time dashboards had cut reporting time to practically zero.
Test execution status and defects raised due to automation highlighted the value of the automation’s ROI.
The Client is a renowned American broadcasting company that provides satellite radio and online radio services operating in the United States. This satellite radio provider upped its game with a new audio platform that focuses heavily on streaming content to a variety of vehicles from many different car manufacturers.
This new platform gives listeners access to more channels, including exclusive streaming channels. In fact, there are more than 10,000 hours of recorded on-demand content available, delivered via the same kind of internet data connection used by smartphones.
Enhanced personalization is another key component of the new tech. It provides curated audio recommendations based on your listening history and selected favorites, by setting up individual listener profiles for different users.
To Solve a Problem You Need to Understand the Problem
Qualitest has been partners with the Client for 10 years and our engineers have been involved across various teams to maintain the quality standards across their programs and systems.
For the launch of this new audio platform, which would improve the Client’s UX as well as offer more and better features, there was a need to thoroughly test various system integrations in the supply chain, from end to end. This raised the need to automate the system integration regression testing.
Qualitest had built out automation using Selenium, resulting in the ability to run ~400 test cases daily. Previously, the team had to run the automation manually resulting in lost productivity.
However, with the new automation in place and teams working in diverse operating models, there was a need for consolidated, easy to understand, real-time reporting across various activities.
The Client wanted to achieve an easy way to gauge the progress of the release and its readiness to launch. To achieve this efficiently, the following activities had to be planned and executed:
- Establish a standard process and discipline of leveraging the tools to record all test activity across onshore/offshore teams for reporting readiness.
- Create daily, weekly and monthly reports showing a consolidated view of projects and releases across defect management and testing tools.
- Showcase the automation coverage and benefit to top-level stakeholders.
- Highlight quality insights across various releases to understand where the majority defects have been detected, which environment and their root cause.
- Achieve team and member wise visibility into the daily and weekly activities.
“There was a need to thoroughly test various system integrations in the supply chain, from end to end. This raised the need to automate the system integration regression testing.”
Real-Time Insights = Real-Life Value
Qualitest leveraged its analytics product, Qualiview, to integrate with the Client’s tools to build real-time dashboards that would serve the Client’s specific needs and save the time of manually creating these.
We implemented an iterative process model, with two clear, well-defined phases:
- Phase 1 – Data Maturity: Processes to mature and structure the data stored in the tools, like ALM and Jira.
- Phase 2 – Dashboard Scripting: Ingest the data, conduct data mapping and transformation as needed and build the dashboards.
Phase 1 – Data Maturity
In addition to our product, Qualitest offers consultancy around data organization and its readiness for maturity. We identified processes to be implemented to capture the data to support the metrics. This included configuration of the tools, setting up workflows for data entry and nomenclature to be maintained for releases.
In parallel, the Client stakeholders were consulted for their reporting needs. Workshops to identify key metrics and KPIs were conducted with executives and leads to finalize the scope of reporting.
Phase 2 – Dashboard Scripting
Once the tools and data configuration were ready, specialized data analysts from Qualitest extracted the data and conducted various data modelling and transformation activities to prepare it for visualization.
This included data validation, verification and testing too. The data was then utilized to create business views in the Qualiview dashboard, which involved reconfiguring the standard KPIs and/or building custom KPIs as per customer needs.
Various automated real-time dashboards were created for:
- Test execution status and defects raised due to automation highlighting the value of automation towards ROI.
- Historic trends in automation execution, automation stability across projects and defect detection rate for advanced analysis.
- Trends highlighting reasons for failure help identify problem areas to increase automation efficiency.
- Functional testing efforts for varied projects across different test cycles in an easy to view layout.
- Cross-project metrics highlighting projects with higher defect detection rate and testing activities.
- Agile metrics, such as sprint reports and burn down charts.
- Team utilization dashboards highlighting team member efforts, including estimated time vs spent time.
“Manual reporting was automated, reducing reporting time to practically zero and increasing productivity of senior team members.”
Qualitest met the Client’s reporting challenges and created the alignment and visibility needed to drive business decisions and strategize testing.
- Manual reporting was automated, reducing reporting time to practically zero and increasing productivity of senior team members.
- ~$1,500 cost savings per week due to improved time efficiency – Saved 19hrs per week improving productivity and time efficiency.
- Achieved an automation coverage increase of 93%, which enabled finding more defects faster across different environments.
- Deeper insights and analysis into historic data guiding testing efforts.
- Increased transparency within the company and especially for senior management.
- Better maturity in test and defect management for teams.
As a result, the Client was able to successfully launch its new audio platform with confidence in its quality.
“Qualitest leveraged its analytics product, Qualiview, to integrate with the Client’s tools to build real-time dashboards that would serve the Client’s specific needs and save the time of manually creating these.”
download the pdf