An expert fact-based guide to making the crowd work for you

Crowdtesting is NOT a commodity when the provider carefully assesses and categorises the skills, capabilities and devices of its software testers. You’re not getting undifferentiated resources, you’re getting experienced and proven testers that a reputable provider knows they can rely on for fast, accurate results.

To guarantee this quality in our crowdtesting, we examine testers’ skills before they can join our community and we moderate their work on the job. We collate and present clear, actionable results through our testing portal so you can scrutinise them at any time and we manage the crowdtesters to ensure they’re delivering the insight you need at the pace you need it.

Unique to our approach at Digivante, we transparently share our crowd testing processes, results, and client testimonials, ensuring you choose a crowd testing platform that meets your specific needs.

#2 There’s a big difference between crowdtesting and automated testing (and it’s not just about cost)

Automation has revolutionised processes in manufacturing, software and services, delivering speed, productivity, efficiency and accuracy and often cutting costs compared to a human workforce. But it’s a myth that every process can be satisfactorily automated.

Many aspects of QA, UX and CX testing are designed to investigate the real human experience of interacting with your digital resources. Automated testing can be excellent for certain tasks, when you’re checking for predictable flaws and bugs or ensuring that defined user pathways and processes are working as designed. But users don’t always follow the pathways you planned nor are they always equipped with the knowledge, experience and skills you expect.

The Digivante crowdtesting community is key to our agile, hybrid approach that blends automation with real-world human experience. It’s a necessary perspective because, after all, your end users are real people.

Crowdtesting allows you to put real human users on the case. You can instruct them to carry out a task on your app or website and specify the precise process steps to see if the exact journey’s outcome is as expected. You can get many users to carry out the same process using different devices and browsers to see if they all have the same experience. You can even ask testers to carry out unstructured exploratory testing, where they follow their own preferences and instincts in the way that your customers and users will.

Automated testing is another service we offer at Digivante, but it’s not a substitute for crowdtesting. When customers give us a testing brief or requirement, we recommend the best services and approaches to deliver the outputs and experiences they need. Crowdtesting meets some testing needs exceptionally well: a high calibre QA partner will be able to explain how, where and why it’s best applied to your project, and the same applies to testing automation too.

#3 Third party crowdtesting has advantages over internal testing

Your own testing team is proven, skilled and professional. You know the people and you know their commitment to your business and QA goals. They also know your products and platforms inside out. These are the justifications we often hear for keeping user testing in-house. While the resource is finite, surely your own team is best placed and most highly motivated to hunt down issues?

It’s absolutely true that your own trusted team knows their stuff – the problem can be that they know it too well. When you’re getting ready to launch a new release or digital product, you’re about to unleash it on a whole new audience who haven’t been involved in the development.

Objectivity and a fresh eye amongst your testers are worth a great deal at this point – well-briefed, professional crowdtesters supply this and will work with commitment and focus to check your team’s work. Their scrutiny gives you peace of mind that you haven’t missed anything. Using a third party takes the political impact out of this: rather than setting teams against each other internally, you’re using an objective, external partner to deploy their specialist skills to help deliver the best result.

Moreover, your in-house team is unlikely to have the range of devices, languages, locations and browsers at their disposal to test digital products across all possible user combinations. Even if you have plenty of budget to buy them, they’ll soon go out of date. Again, the large scale of the crowdtesting community comes into its own: your provider can access people with any type of technology or skillset from their pool of trusted people, including the latest devices.

#4 Crowdtesting does not mean outsourcing

Bringing any third party into your IT team can create unease. In a sector where contractors and outsourced providers are familiar sights, it’s not surprising if your team is wary of partners who appear to be coming in to take over their jobs.

Outsourcing means giving over existing work in your business to an independent organisation to manage in its entirety. It’s also sometimes referred to as offshoring, and can have negative connotations where workers see work they were doing has been passed to cheaper workers in other countries.

Crowdsourcing on the other hand isn’t necessarily passing work to one single entity, though in the world of crowdtesting you are working with a single entity that helps you manage the crowd projects. But in this instance, you’re not giving over your work in its entirety. Rather you are using crowdsourced testing to support existing QA efforts.

Using crowdtesting is liberating for your development and QA teams. Because it’s an on-demand service, you can switch it on to handle peak requirements and switch it off when the need isn’t there. Using crowdtesters to handle a high volume of testing in a short period of time gives your teams the rapid QA results and insights they need to correct errors and refine digital products before launch, to optimise the customer or user experience and maximise the positive impact of their work.

When your team isn’t on the hook for user testing, they can focus on higher value internal work and on rectification. They’ll also be released from the pressures of surges in workload, when testing needs to take place at night or over the weekend. Global crowdtesting communities are available to work 24/7 and can ramp up rapidly to deliver weeks of testing in just a few hours.

#5 Crowdtesters are extremely reliable (as long as they’re properly managed)

We often hear the accusation that crowdtesters routinely fake or duplicate errors because they’re paid on the basis of the number of issues they uncover. This might be the case for some crowdtesting providers, or it might historically have been an issue, but it’s not something we’ve seen a lot of in practice.

For one thing, crowdtesting firms wouldn’t stay in business long if they developed a reputation for inaccurate QA testing and creating more work and complexity for their clients in a time-pressured environment. For another thing, crowdtesters who work for high calibre service providers usually have their own individual reputation and ranking to think of. They’ll often be reviewed and scored after a project: they want to have a high rating so they’ll be selected for more work and given more responsible and interesting tasks.

Digivante testers work up to a top rating by finding issues that are accepted by our team. To get a Elite rating , testers must have achieved 200 accepted defects, an overall acceptance rate of 70% and be consistently active in the portal.

Before they even get to that point, our crowdtesters have been assessed on their knowledge and experience. Their work is monitored during a project, to make sure they’re not reporting duplicate or insubstantial errors, either by accident or deliberately. Our test reveiwers check quality and oversee the results to make sure they’re clean and accurate. They investigate and give feedback to testers and if necessary, exclude them from the community if their work isn’t up to standard.

Digivante crowdtesters are paid to be accurate and detail-focused. They’re only rewarded for discovering validated errors.

#6 You can trust reputable crowdtesters with security and confidentiality (as long as you choose wisely)

We hear a lot of doubt about this, especially from clients working in highly regulated industries like financial services. Of course, with GDPR and other data protection regulations more stringent than ever, confidentiality and data security are a concern for every organisation that holds customer data.

There’s also the issue of intellectual property and competitive information. If you’re about to launch an innovative app or leading-edge functionality into a fiercely contested marketplace, you don’t want to risk news of it leaking out to your competitors. An uncontrolled crowdtesting community would certainly pose a risk in this situation.

Choose your crowdtesting provider wisely and ask them for evidence of their security controls and the assurances they make to their clients.

At Digivante, every crowdtester signs an NDA before commencing a project and their work is monitored throughout.

Testing platforms and processes are key to maintaining compliance, security and confidentiality. Leading crowdtesting firms can carefully control testers’ access to systems, making sure that they can use and explore only the designated pages or sections. They use secure online systems to share testing projects and tasks and to report test data securely back to a central platform.

#7 Crowdtesting creates more evidence but not necessarily more work

Receiving a large volume of raw data from test results could put development and QA teams under pressure to analyse and act on it. Like all data, in its raw form, testing feedback doesn’t necessarily add value. It needs interpreting. If your in-house resources are stretched already, the last thing you need is that heap of extra work.

That doesn’t have to be the situation. Experienced crowdtesting providers who understand their clients’ needs have thought this through; they add value by checking, processing and analysing test results for you. Instead of thousands of test results, you get clear, outcome-oriented and actionable reports that point your team towards the priorities for remediation and improvement.

Digivante’s software testing portal gives our clients direct, real-time access to test results in formats that they can immediately understand and use. Working to an agreed brief, we can make sure the test activity and results are focused on specified areas of your user experience or digital functionality.

Despite the name, crowdtesting doesn’t involve the whole crowd. You’re always in control of the test volume. Applying a very precise definition of skills, devices, operating systems, localities and user demographics to replicate target user experiences as closely as possible, will set the quantity of testers on a project. By focusing and refining your testing specification, you’ll get focused and specific results that won’t create an administrative burden of processing and interpretation.

Don’t make assumptions about crowdtesting

Of course, crowdtesting doesn’t solve every QA problem all by itself, but the most successful organisations know its value in maintaining digital quality standards and creating excellent user and customer experiences. They also know the value of a trusted partner who can build crowdtesting into a customised QA solution and offers a range of digital testing services.

Crowdtesting is a powerful and valuable discipline in achieving digital quality, as long as you can harness the power of the crowd and focus it precisely and carefully on achieving your testing goals. The crowd as an uncontrollable rabble is a myth. Crowds have the power to deliver enormous value in testing, but it’s crucial that there’s there’s stringent quality criteria and vetting in place to join a community, as well as strong management processes and security protocols for carrying out the testing.

The four conditions that characterize wise crowds: diversity of opinion (each person should have some private information […]), independence (people’s opinions are not determined by the opinions of those around them), decentralization (people are able to specialize and draw on local knowledge), and aggregation (some mechanism exists for turning private judgments into a collective decision).

James Surowiecki, The Wisdom of Crowds: Why the Many Are Smarter Than the Few

With crowdtesting, you get access to resources and environments that you can’t replicate internally. You can dramatically speed up testing time and crucial moments in your development and digital product launch cycle. Test monitoring, reporting and analysis can all come as part of the crowdtesting package. Choose a partner that has a secure and user-friendly testing platform, so you get results fast and cut down the administrative burden for your team.

A knowledgeable and experienced crowdtesting partner like Digivante can help you create testing briefs that deliver high-impact results and can rapidly marshal the precise resources you need from a trusted crowdtesting community.

If you’d like to find out more about our crowdtesting services, get in touch.