Un-beta-able – what’s better than alpha-beta testing?
You’re probably familiar with Alpha Beta testing by now.
After all, the terminology has been floating around since the 1950s – back when IBM started A and B testing their ideas. And although the computing giant isn’t necessarily the tech leader it once was, this type of testing’s become the standard for internal development teams ever since.
The thing is, it may have been around for over 70 years, but many businesses are still struggling to strike the right balance between the two.
All too often, Alpha Beta testing can become a bit of a headache. These time-consuming tasks are usually piled on the overflowing plates of project managers or product owners. The problem is, they’re unable to define the requirements and ensure valuable outcomes from this type of test. That’s because they’re under enough pressure to release their products prematurely – without having to define new processes in an unfamiliar area. Not good.
OK, but what’s the alternative?
Well, before we get into that, let’s get one thing out the way first…
What are the key differences between Alpha and Beta testing?
Alpha testing is the very first phase of validating whether or not a new product or app is performing as expected. It’s carried out early in the development process by internal staff and can be as big or small as you like. Crucially, the feedback comes from within the company.
This is usually followed up with Beta tests, where a sample of your audience outside the business tries the product out. Unlike those preliminary Alpha tests, this process is one of the final steps before going live. Essentially, it’s the last chance to ensure your intended customers are satisfied with the experience before it goes out in the world.
Often businesses will ask for real-world volunteers in order to get feedback. For example, you can join Snapchat’s Beta to test new features before they go live. If they’re in the right demographic, members of Snapchat’s Beta can earn $10 per test.
And there are plenty of advantages of running Alpha and Beta tests for businesses, too. But as you’ll soon discover, this isn’t a bulletproof process.
What are the advantages and disadvantages of Alpha Beta testing?
Advantages of Alpha testing
Alpha testing incorporates both black box and white box testing. That means it’s designed to put both input and output functionality through its paces, while also checking the system’s design and internal structure. Importantly, it does so in a simulated environment made up of relatively realistic testing conditions. That means it’s close to the real thing whilst still being a safe space to test before being released into the wild. And as these are internal tests, the turnaround for results is usually fast.
Well, in theory anyway.
This snappy turnaround means dev teams can jump on and, hopefully, resolve any issues before entering the Beta phase.
Advantages of Beta testing
Beta testing not only offers an additional chance to catch bugs before release, but it does so in a real environment. It’s carried out by a much larger group of testers than a business’ in-house QA team. The idea is that this should increase scope and representation. It also frees up internal development and testing resources, saving project time and developer budget. Finally, there’s usually an incentive to test. It could be financial, early access to new features or simply getting bragging rights that you “tried it first”.
So, you’d think by running a combination of Alpha and Beta tests the picture would be crystal clear – right?
Well, not necessarily.
Disadvantages of Alpha testing
You could argue that one of the benefits of alpha testing is also its biggest drawback: the fact that it’s run in house.
But, why when these folks know the product.
Well, that’s it exactly.
Development teams and other internal stakeholders might have a vested interest in whether a product “passes” the test or not. And if developers are involved in the testing, they might be hesitant to point out something they’ll be responsible for fixing. Alternatively, they might not want to upset the apple cart if they find an issue that a colleague has caused.
Essentially, whether they’re conscious of it or not, their views are unlikely to be unbiased.
But that’s not the only problem.
If the dev team’s working in agile sprints, there will always be new and more pressing priorities – it’s the nature of the beast. So this makes it hard to timebox an activity like Alpha testing, especially if it isn’t something they handle in their day-to-day.
With those factors in mind, the best way to run an Alpha test is to do so with a department or team that has nothing to do with delivering the product.
But will they have the time or urgency to prioritise the task? It’s debatable.
Also, doesn’t outsourcing this activity to another group sound a lot like Beta testing? Albeit on a much smaller scale. If that’s the case, it’s hard to see the value in it at all.
Disadvantages of Beta testing
As we mentioned above, many Beta testers will opt-into the process because they’ll be compensated in some way. However, they’re not on the payroll and the incentive is usually fairly small.
This means testing can feel like a bit of a thankless task. Especially if your internal testing team needs to ask them to try and reproduce the error.
“I didn’t sign up for this – I’m too busy now – I don’t remember how that happened,” are all common responses. That’s if you get a response at all.
And after a few days of this, many testers may become demotivated, lose interest and – worse still – drop out. If that’s the case, businesses may end up with a relatively small and unrepresentative pool of feedback.
Remember, if your product isn’t as known or beloved as Snapchat or Playstation, it’s unlikely you’re going to get the same level of feedback. These guys are joining the Beta because they already love the product.
Unfortunately, it’s unlikely what you get back is going to be chapter and verse.
But the feedback you do get has still got to be both clear and unemotional. It’s not enough for them to say they “don’t like how it looks” – why don’t they?
Unfortunately, the line can easily become blurred between functional and usability issues. This can easily be controlled in moderated UX tests – but not on the scale of beta testing.
And if the feedback’s no good?
Well, it’s back to the drawing board.
A disaster if the management team is expecting a product launch post-beta.
Alpha Beta testing gone wrong – a cautionary tale
A client is using an audio player with their app. They suddenly notice that the player’s timer isn’t changing during playback.
Weird.
Especially, when the app had been tested internally for eight weeks and in public Beta for a month.
So, what went wrong?
Well, the internal team missed it because they were too focused on other issues. And the Beta group didn’t raise it as they assumed it’s the way it worked. After all, how could something like this make it through two months of internal testing otherwise?
The lesson here is that if you’re going to do Alpha and Beta tests, you need to ensure two things:
- You have a clear set of objectives
- You have direct contact with your testers
These are so important but are too often overlooked. Some app developers even make the mistake of running their Beta on app stores, gathering “feedback” from reviews.
OK, there are many problems with this approach but the big ones are:
1. Put out a bad Beta trial and you might never go live.
Just think, the reviews are out there for all to see – so what if they’re bad? The reality is your app might have run a tad slow on one device. But unfortunately, those first impressions are everything.
2. Feedback through reviews.
We’ve already established that chasing feedback from beta tests can be a chore. But feedback through reviews? Good luck to the person who has to sift through 574 comments to find those “make or break” issues.
Go too early with too wide an audience and get things wrong and they’ll never give your product a second chance.
Unfortunately, there are no shortcuts in this process.
So, what’s the alternative?
Our approach to un-beta-able A and B testing
Looking for a foolproof approach to testing? One that takes into account the views of real users and guarantees customer satisfaction. Well, follow these two steps and you’ll never have to worry about the frustrations of Alpha Beta testing again.
Avoid the Alpha and go exploring
You could run an Alpha test in house. If so, it might go a little something like this…
- Find the internal resource in house and reprioritise their workloads
- Get them to review your product’s design specification and functional requirements
- Ensure they develop a thorough test plan and test cases
- Plot out the time to execute this test plan and ensure they log any defects you find
- Schedule the bugs to be fixed into your agile sprint – shuffling around existing tasks
Sounds like a lot to plan out and manage? It’s not going to be as quick as you thought.
Instead, you could stick to the plan with that busy product release and book in some outside exploratory testing.
An experienced pool of global professional testers can quickly find any weak points and usability issues that might be buried deep within your product. What’s more, they’re real users who are completely unbiased. That means they come with no preconceptions about how your product “should” work. Plus, you’ll get the feedback in days – not weeks, and it’s all summarised in clear reporting that allows you to see the exact issue.
In fact, it’s reported that 82% of companies now use exploratory testing as a software testing methodology. (PractiTest) – so you’re not alone.
Looking for a little more info? We’ve written more about exploratory testing here.
Beat the Beta and find hundreds more issues
Exploratory testing works best when you run it alongside other types of testing. This is because it’s designed to highlight issues – as opposed to confirming consistent performance.
So instead of carrying out Beta testing by itself, why not run an exploratory test from a functional perspective and combine it with some usability testing?
Not only will the exploratory test uncover hundreds more issues than Beta testing; UX testing results in more detailed feedback from real users. On top of that, Digivante will work with your Beta group. We can create a faster and more accurate process by using our portal, gathering and prioritising issues accordingly.
Want to use our crowd testers instead? You can rest assured they bring with them an unbiased approach. And it’s a representative group too. Because at Digivante, we draw on professional testers from 149 countries around the world – meaning they’re available around the clock. Crucially, we can deploy 200 testers per test, delivering 90 days of testing in just 72 hours.
But our crowd are not only testers, they offer a true representation of all the devices, browsers and OS your customers are using. And with cross-browser testing, we’ll check your site or app across a minimum of 25 device and browser combinations. If they’re using it, we test it.
With over a decade of finding bugs, we’ve seen just about everything that could go wrong with an app or website. Consequently, we’ve developed an eye for spotting issues quickly, efficiently and before they escalate.
So, isn’t it better to seek help from professional testers before Alpha or Beta tests go awry? Otherwise, leadership might be scratching their heads as to why the pros unearthed an issue in two minutes that took internal devs two months to find. Just a thought.
Have you got a pressing software product release you need some help testing? Get in touch.
