Our Client is a household-name UK footwear and apparel brand founded more than 70 years ago. For our Client, as for most retailers, Black Friday is one of the single most important sales days of the year and sets the pace for the all-important holiday shopping season. Intense brand competition via deep discounts and special offers drives consumer demand far above normal levels and can create spikes, surges and a punishing load on systems in every channel.
To maximize Black Friday’s revenue potential and gain a competitive edge for the whole holiday season, our Client knew they had to deliver a fast, error-free end-to-end customer experience.
Adding to the usual holiday readiness challenges, this particular year our Client was in the middle of critical system changes. They were upgrading to the latest release of MS Dynamics 365, incorporating performance fixes to address outages and other issues that had occurred during the same period in the previous year. Also, they had recently implemented Azure integration services into their ERP environment.
They needed assurance that their upgraded ERP system would perform flawlessly for Black Friday, with last year’s issues fixed and all other flaws, bugs or bottlenecks identified. At the same time, they wanted to know if their newly integrated systems could handle substantial future growth.
The Client turned to Qualitest to manage and deliver testing services that would validate their integrated systems and provide technical and business assurance of quality performance across the Enterprise estate.
Qualitest created a comprehensive test plan that started with securing stakeholder consensus and included rigorous performance testing end-to-end and at component level.
Consensus-building: We held multiple workshops with business and technical SMEs to determine the critical nonfunctional requirements (NFRs) related to performance and agree on the necessary scope to represent the forecasted peak usage for Black Friday, plus 50% future growth.
Strategy: We created a detailed performance test strategy that included selecting appropriate test types, environment and workload over the integrated ERP system and monitoring mutually agreed-upon acceptance criteria.
Execution: We performed peak volume simulation and exhaustive performance testing for all customer and business user channels, with various performance test scenarios aligning to the workload model forecasted for a peak day.
Automation: We set up a performance test regression suite to assure future releases, developing a robust, reusable automated test framework that would cover the agreed-upon set of business process activities and supporting data flows through interfaces.
Tools: We used AppInsights, PRTG Network Monitor and to observe & analyze metrics, logs and traces within the end-to-end integrated systems.
Our solution enabled significant improvements to the Client’s system for the upcoming holiday season and assured system readiness to meet their goals for future growth.
We also identified multiple performance bottlenecks in the Azure integration services (Azure APIM and Azure Queues) at peak load, potentially frustrating customers with issues such as fulfillment errors, slow-loading pages, and increased response and invoice processing times.
For example:
