Crowdsourced testing consists in placing an application in the cloud, to be tested by a community of testers.

This is a very different way to test than the traditional testing. You can do a lot of tests on crowdsourcing, such as safety, performance, localization or usability tests, but the most common are functional tests.

In a traditional environment of software development, waterfall or agile, the tests start at the same time that you start to draw the solution. A test analyst collaborates on the design and construction of the product, from its initial phase, along with the business analyst and the remaining development team. It is critical that the test analyst understands well the requirements of the client, both functional and system, to start drawing a test plan. The environment in which the product will run, the market or the type of users is important data to identify the potential problems and challenges that the product will be subject to and thus be able to strengthen the tests. Knowledge of the project plan is also important in order to manage the test activities in each of the phases of the project.

We can do tests indefinitely. In fact, the test work is never finished in the sense  “everything is tested” because we can always do more and more, and multiply the 1001 variables. Anyway, it could be an endless job. But since this is neither feasible nor desirable, the test analyst will have to define the testing strategy and incorporate that into the development project plan.

In a crowdsourcing environment, focusing only on functional tests, all this happens in a fraction of seconds or minutes.

Typically, functional tests are done on an application that is in a state of high maturity, that is, has gone through a phase of traditional testing before reaching the crowdsourcing community. Because we are in crowdsourcing, we can have a crowd of testers simultaneously testing the same application.

On the other hand, the time available to test varies between two days and one week, rarely is less than or greater than this range. You can already see that all the phases I mentioned earlier in a traditional system will have to be done in a very short time. There is no time to formally draw a test plan, not even to write test cases. Much less there will be time to automate whatever it is, nor would that make sense in functional tests.

Once the tests are started, you have to face the competition. We are not testing alone, nor sharing a test plan with a colleague, we are testing with a crowd of other testers. We do not know the test cases of the other testers, and often they similar to ours, at least the most basic ones will be. The bugs we found will not be able to be logged if someone has done it before us. Duplicated bug records (in a commercial crowdsourcing environment) are not allowed. So it can be frustrating when we detect a strange behavior in the application, repeat all steps until we find the pattern, and when we are ready to start logging the bug founded, we find that someone has done it before.

Sometimes we can do great tests, be able to identify important issues but nobody will ever know!

tomate cereja

So speed is critical in this type of tests. First we have to be quick to accept the challenge, if we take too long, the others have already begun to test. Second, quick to understand well the scope of the functionalities that are intended to be tested, to know well these features and to draw a test plan (this in the case of exploratory tests). Third, fast running the tests. Fourth, super quick to log a bug found. Finally, we have to know when to stop.

This type of testing can be paid for or can be done just for fun. As paid work there are two types of most common modalities: payment for the work done, that is, time spent executing a test plan provided, or payment for registered bugs, according to type and importance to the client. Then there are the competitions, also with a limited time, where there are final prizes for which more and better bugs have registered, and the platforms or organizations that pay for each bug of extreme importance found. In the latter case we are not talking about limited periods of time. There will be other modalities, of course, but these are the most common.

In crowdsourced testing, in the case of functional tests, the most common is to launch a test cycle that has a limited duration and testers that match the intended user profile are invited . These cycles can be for testing web or mobile applications and can be specific to certain geographies or certain languages. The test environment is very specific, that is, the type of mobile device, the operating system, the version or even the version of a browser. The type of tests can be exploratory or the execution of predefined test cases. The bug log is very demanding, that is, the conditions, the steps to run and the documentation with logs, images and videos are indispensable.

It should be noted that crowdsourcing testing has a different purpose than traditional testing.

The advantages for those who use this type of testing are several, among them the possibility of testing an application, before launching in the market, in many real environments that could never achieve in virtual and simulation environments. It is possible to adjust the cost and duration or depth of the tests in order to achieve a balance between cost and benefit.

For those who perform tests, this modality allows them to practice test techniques in a demanding environment with strong competition. Another advantage is the discipline that is practiced in the way a bug is logged. The way a tester describes errors found can be a headache for a developer, so this is a great way to practice.

In short, these tests do not replace any type of tests during the development phase. They should be seen as a complement, as something that comes to ensure a high level of knowledge about the quality of our application, in any aspect. For those who perform are an excellent training and improvement tool.

Have you tried it?

(picture: cherry tomato from our kitchen garden)