Ecommerce QA & Release Management Case Study
Software Quality Engineering
Software Release Management
About the Client
Client is a purveyor of luxury design, refinement, and elegance within the high-end lifestyle industry. Their product range encompasses furniture, lighting, textiles, rugs, bathware, decor, and outdoor items, alongside offerings for infants, children, and teenagers. Client is a distinguished American home furnishings company headquartered in Corte Madera, California. The company distributes its products through a network of retail stores, catalogs, and online channels.
Key Data About the Client:
What the client has to say?
“ Progile Tech comes highly recommended. Their deep understanding of our challenges extends beyond surface issues, as they employ a methodical approach to uncover and address root issues. They have developed a robust QA Governance Strategy and an execution plan to increase the frequency of our production deployments, all while prioritizing deployment stability. Their ability to swiftly adapt to our environment and deliver immediate value has been impressive. With Progile Tech as our QA partner, we have full confidence in their capabilities to ensure reliable releases.”
Sudip R, VP of eCommerce Engineering
The Scope of the Work
Clients online store and an integrated application called Concierge play a pivotal role in Client's annual revenue, contributing to approximately $3 billion each year, accounting for 85% of the total revenue. The online store serves as the client's self-service platform for shopping and making purchases, while the Concierge application is utilized by gallery and store associates to curate customer shopping carts and facilitate checkouts. As the Client is currently undergoing rapid growth and expanding its presence internationally, the company aims to increase the frequency of production deployments to achieve daily releases.
However, historical deployment practices have been marred by production issues, resulting in frequent rollbacks, emergency fixes, and occasional production downtime, ultimately leading to revenue losses. In response, the engineering leadership team recognized the need to establish robust QA and Release management structures in order to transition to a daily release cadence and minimize production disruptions.
To address these challenges, Client has enlisted the assistance of Progile Tech in the following areas:
- Create a lightweight QA Governance and accountability structure and strategy encompassing manual, automation, and performance testing.
- Implement release management processes for planned releases, hot fixes, and deployments no longer requiring senior leadership approvals.
- Provide resourcing and capacity planning to facilitate the shift from infrequent releases to daily releases while ensuring thorough regression testing and adherence to the "golden path."
- Foster the maturation of cross-functional collaboration among Program Management, Engineering, Product Management, QA, and other business teams to enhance overall efficiency and effectiveness in the release process.
Our Solution
At Progile Tech, we recognized that client's eCommerce environment is highly distinctive, marked by its technical complexity and geographical dispersion of teams. These factors have resulted in challenges related to communication, accountability, and test coverage, ultimately contributing to unstable production deployments. Additionally, the teams regularly contend with leadership escalations and ad-hoc initiatives, further complicating the already intricate deployment processes.
Given this understanding, Progile Tech's initial step was to deploy a Senior QA consultant to conduct a comprehensive assessment of the existing structure, organizational setup, and processes. The goal was to develop a strategy and a set of recommendations. Once the recommendations were formulated and accepted by the client, Progile Tech proceeded with the following objectives:
- Established a lightweight QA governance framework that comprehensively revamped the existing QA processes. This involved alignment on test procedures, resource planning and supervision, the adoption of test management tools, and the formulation of a framework and maturity roadmap for automation and performance testing.
- Enlisted and integrated a team of 20 professionals, including onshore and offshore QA team members such as an Onshore QA manager, offshore QA Managers, onshore QA leads, onshore and offshore manual QA members, Onshore Automation Leads, and Onshore Performance Engineers.
- Instituted a release management governance process encompassing branching/forking strategies, pre-release/cutover activities, Change Advisory Board (CAB) processes, post-release merge processes, and communication across team members.
- Introduced two Release Managers, one onshore and one offshore, to enhance release management capabilities.
- Implemented a 24/7 testing resource availability rotation program to address ad-hoc and emergency releases promptly.
- Introduced daily testing check points and hand-off meetings involving both onshore and offshore teams for seamless coordination.
- Adopted Test Rail as a test management tool and migrated approximately 5000 test cases from scattered locations to centralized repositories for better organization and run history.
- Automated 75% of golden path testing using the Selenium UFT framework.
- Developed performance testing execution plans and implemented a shift-left strategy to commence performance testing during the build cycle rather than after build completion.
Outcome:
Due to the dedicated efforts of Progile Tech, Client has established a solid foundation that allows them to scale up to five releases per week. These releases are now subject to thorough regression testing and receive approval from release managers and relevant stakeholders. Moreover, Client has successfully automated 75% of all regression test cases with a confidence level of 95%.
- Transitioned from manual testing to an automation-first approach.
- Reduced Hotfix Regression testing time by 75% with automation (what used to take 3 days is now completed in less than 1 day).
- Expanded the testing scope to include Usability, Security, and Accessibility, ensuring compliance with InfoSec and WCAG standards.
- Enhanced user experience by broadening test coverage across Chrome, Safari, mobile devices, iPads, and other browsers and devices.
- Fully transitioned to automation for upcoming releases, streamlining the process.
- Centralized test execution and traceability, moving from scattered Excel-based testing to a more efficient, streamlined Test Management Tool (TestRail).