Crowdsourcing your software testing can be a great approach for some aspects of the testing lifecycle. By employing the crowd for processes like beta testing and compatibility testing, your application is presented to a wide variety of devices over a wide variety of connectivity and network standards across the globe. This can give you a great idea of how your application will do “in the wild,” or in the hands of actual users.
However, for a complete, exhaustive testing process, crowdtesting alone is not enough. It’s a great option for the final stages of software testing, but these are not the only stages in which a product should undergo testing. As we tell clients all the time, testing should extend throughout the development process, not only once an application is “done” and ready for release. Fixing small defects in the initial phases of their life cycle is both easier and cheaper than trying to fix bugs which have had a chance to develop alongside the application. Bugs that have been present for an extended period of time often have the system designed around them; removing bugs at that point is a lengthy process, as it could mean rewriting huge portions of vital code.
Crowdsourcing is performed by people who represent the common user, and that’s why it gives such a good feel for how the application will fare in the wild. They test the few pieces of the application that a user would actually use, and how they would do so. However, that means that the testers actually have access to only a small part of the application, the front end/UI, and have little to no interaction with the back end or how the application itself works. When you take a group of dedicated testers and give them a list of requirements, the result is an exhaustive testing process which looks at how the product will be used by one individual as well as how things are working behind the scenes. The crowd can only test basic, common business processes, not the entire system – further bugs could be hiding everywhere else.
Crowdsourcing alone may not give the best feedback on applications; instead, use a holistic approach that blends professional testers with the crowdsourced community.
Many sources say that an inclusive, holistic approach for testing is more fruitful than sticking to just crowdsourcing. This holistic approach brings a variety of different methods and methodologies into play during the testing process, and in general tends to be highly praised among testing professionals. Applause.com, a resource compiled to promote crowdtesting, agrees that crowdsourcing alone may not give the best feedback on applications, and instead recommends a holistic approach that blends professional testers with the crowdsourced community. They speak out in favor of a diverse testing approach that pools these two resources, saying, “Having this diversity of staffing allows you to scale your resources up and down in a fluid manner, meeting tight deadlines during peak periods of development and testing, while controlling costs during slow periods.”
At QualiTest, we agree wholeheartedly that crowdsourcing alone will yield unsatisfactory results for any process besides beta testing. We’ve found that the best way to use crowdsourcing is to integrate it as one of those models, but certainly not as the only one. Rather, the most successful project is one which uses seasoned testing professionals, outsourced or in-house, before the beta version is even complete. This approach is a much better guarantee that the fundamental bugs have been addressed and the hidden defects have been identified before the product even goes out for crowdtesting, ensuring much more reliable, actionable results from the crowd’s experiences.
