Rigorous Holiday Readiness Testing Helps Assure a Perfect Performance for a Famous UK Footwear Brand
We developed a regression test suite and a robust, reusable automated framework to assure future releases, and systems were assured performant at peaks of 12K/sales per hour with 100% data persistence.
Recent ERP updates, integrations and performance fixes were untested as Black Friday and the holiday season approached.
Client needed assurance their upgraded systems could handle forecasted heavy loads and be performant at additional 50% future growth.
Exhaustive performance testing measured upgraded systems E2E and at component level.
We developed a regression test suite and a robust, reusable automated framework to assure future releases.
Found 10+ bugs and identified multiple Azure-related potential bottlenecks and root causes.
Systems were assured performant at peaks of 12K/sales per hour with 100% data persistence.
Our Client is a household-name UK footwear and apparel brand founded more than 70 years ago. For our Client, as for most retailers, Black Friday is one of the single most important sales days of the year and sets the pace for the all-important holiday shopping season. Intense brand competition via deep discounts and special offers drives consumer demand far above normal levels and can create spikes, surges and a punishing load on systems in every channel.
To maximize Black Friday’s revenue potential and gain a competitive edge for the whole holiday season, our Client knew they had to deliver a fast, error-free end-to-end customer experience.
Ensuring exceptional performance under peak pressure
Adding to the usual holiday readiness challenges, this particular year our Client was in the middle of critical system changes. They were upgrading to the latest release of MS Dynamics 365, incorporating performance fixes to address outages and other issues that had occurred during the same period in the previous year. Also, they had recently implemented Azure integration services into their ERP environment.
They needed assurance that their upgraded ERP system would perform flawlessly for Black Friday, with last year’s issues fixed and all other flaws, bugs or bottlenecks identified. At the same time, they wanted to know if their newly integrated systems could handle substantial future growth.
The Client turned to Qualitest to manage and deliver testing services that would validate their integrated systems and provide technical and business assurance of quality performance across the Enterprise estate.
A test plan to stress out flaws, bugs and bottlenecks
Qualitest created a comprehensive test plan that started with securing stakeholder consensus and included rigorous performance testing end-to-end and at component level.
Consensus-building: We held multiple workshops with business and technical SMEs to determine the critical nonfunctional requirements (NFRs) related to performance and agree on the necessary scope to represent the forecasted peak usage for Black Friday, plus 50% future growth.
Strategy: We created a detailed performance test strategy that included selecting appropriate test types, environment and workload over the integrated ERP system and monitoring mutually agreed-upon acceptance criteria.
Execution: We performed peak volume simulation and exhaustive performance testing for all customer and business user channels, with various performance test scenarios aligning to the workload model forecasted for a peak day.
Automation: We set up a performance test regression suite to assure future releases, developing a robust, reusable automated test framework that would cover the agreed-upon set of business process activities and supporting data flows through interfaces.
Tools: We used AppInsights, PRTG Network Monitor and to observe & analyze metrics, logs and traces within the end-to-end integrated systems.
Our solution enabled significant improvements to the Client’s system for the upcoming holiday season and assured system readiness to meet their goals for future growth.
- We identified and corrected 10 high-priority performance bugs.
- Our teams assured 100% data persistence (zero data loss) under load.
- Our testing assured end-to-end response times are within expected SLAe., under 40 minutes on 5 channels and about 12K sales orders/hour.
- We produced automated reusable assets to enable the Client to meet future goals of an additional 50% growth.
We also identified multiple performance bottlenecks in the Azure integration services (Azure APIM and Azure Queues) at peak load, potentially frustrating customers with issues such as fulfillment errors, slow-loading pages, and increased response and invoice processing times.
- We found the system was unable to create JSON (data) files for warehouse processing.
- Insufficient thread utilization (restriction to 16 threads) made the system incapable of handling sufficient concurrent calls and caused visible spikes in response times.
- The implementation of a third-party patching tool caused high CPU utilization.