While not all companies work to a two-week sprint, it does seem to be a common time-frame we observe the most at Digivante. Regardless of that, all companies share the concern of releases not being ready to go live within the planned time-frames, with implications not only for that release but all future releases in the road-map.
A lot goes into a release with multiple business units playing their part – so much can go wrong. One of the most frequent reasons for delays to a sprint is a bottleneck in testing, and this is without considering that most testing solutions test only on a small number of platforms, often just one or two.
Imagine if they were to test on a range of platforms to ensure the majority of their users get the same experience… Imagine if they were to test on all the key devices, platforms and operating systems their customers use.
To achieve the timescales, many businesses adopt a policy of ‘push and pray’. Hoping to avoid the ongoing impact of a delay to the release, they push it out, without thorough testing, and simply hope for the best.
Automation or manual?
Performing automated testing is the goal of every company we speak to. However, as most have experienced, it does have its drawbacks. While, in theory, automation should significantly speed up the process of testing, in practice there’s an investment of time and effort to get it up and running, and then when it returns a fail it takes human intervention to identify why it has failed.
Another drawback is that it doesn’t allow for a change in business focus; once the script has been coded, if you decide to change focus or add in something, you need to begin all over again. So if you are releasing twice per month, automation can in practice slow down the process, especially in the short to medium term.
That said, it does provide a solution in the long term to regression testing. Over time you can build a robust pack: this creates confidence that the fail results are real issues that need fixing and the pack can be executed thousands of times out of hours without impacting your time-frames.
Manual testing also has its drawbacks. In order to have a team large enough to execute hundreds of test cases on a regular basis, with each of those test cases being executed on a range of platforms, you would need a lot of ‘bums on seats’ and incur the cost of that. Add into the mix the significant amount of downtime, where you are paying a QA team to do nothing, and it’s an expensive business.
It is also a lengthy process which is open to human error as you are typically relying on a small number of people to execute many test cases on multiple devices. This approach is known to lead to ‘browser blindness’ which ultimately leads to false positives.
However, with a large enough QA team, with access to all the platforms that your customers use, manual testing will allow you to quickly and efficiently run both new functionality and regression testing and allow for changes to business focus with almost no delay.
Find out more about why you should manually test your app or website on old devices.
From either/or to both
So if neither automation nor manual testing fits the bill on its own, logic suggests that combining both would be the winning solution. However, in most cases, companies don’t have enough QAs to cover the gaps while the automation is ramping up to being effective.
Here at Digivante, we’ve been working on a solution to this QA challenge, drawing on our global community of professional testers so that the companies testing new apps and websites can enjoy the benefits of both automation and thorough manual testing.
