You are on page 1of 12

Mobile Application Testing

How Crowdsourcing Stacks Up Against Offshoring,

Emulators and Beta Programs for Testing Mobile Apps

A Free eBook from uTest

Mobile Application Testing (or lack thereof)
O f all the challenges that come with producing a quality mobile application, testing is often an afterthought that falls near
the bottom of a company’s list of priorities. But in an industry where even the smallest flaw can ruin a mobile app (and the
business behind it), how can this be? Why are so many mobile app companies ignoring this critical phase of releasing high-
quality apps? And what do they risk by doing so?

The answer is that comprehensive testing of mobile apps has proven to be difficult and expensive using conventional
methods. While the testing matrix for mobile apps has grown exponentially (handset maker & model, wireless carrier, OS,
language, location, etc.), the methods for testing mobile apps have not kept up. They are outdated, inefficient, and above all,
impractical for the world of mobile. Thus, despite the obvious risks associated with releasing a buggy mobile app, the less
complicated, less costly alternative has been to test a little bit and hope for the best. Though this scenario paints a dark
picture for mobile app companies, there are other options.

This eBook will identify four of the most popular approaches for testing software (listed below) and explain why they are
not enough for the world of mobile apps. This will be followed up by a quick introduction to crowdsourced mobile testing.

If you are testing a new mobile app (or a new version), these four testing methods are not sufficient on their own:

• In-House: For reasons of cost and coverage, it is inexcusably naïve to expect an in-house team to provide comprehensive
testing coverage in a reasonable amount of time. Learn why the in-house route works in other areas, but not in mobile.
• Outsourcing: The same problems that plague in-house teams affect outsourcing firms ten fold. If quality and coverage are
priorities for your mobile application, then outsourcing will be a major disappointment.
• Emulators/Simulators: There’s literally a world of difference between actual testing, and testing in a simulated lab. While
there are benefits to leveraging emulators and simulators, ignoring real-world usage is a proven recipe for disaster.
• Beta Testers: The era of beta testers a substitute for comprehensive testing is long gone. Users use. Testers test.

The crowdsourced alternative: Learn how community-based testing, aka “crowdsourcing” is perfectly
suited to meet the needs of today’s mobile application developers.

A Free eBook from uTest Page 1

What’s At Stake? The Challenges of Mobile Application Testing
Users, Bugs and Money: According to a recent study highlighted on TechCrunch, the average shelf life of an iPhone app is
roughly 30 days. It turns out that, for free apps, less than 20% of users return to an app even one day after downloading it. And
by day 30, less than 5% of users are still utilizing the app. And for paid apps, the drop-off is even slightly steeper. Ouch.

Why the drop-off? For starters: bugs - especially ones that appear within minutes of downloading - are likely to cause users to
abandon the application after a short period of use. With the variety of apps now available, users have an extremely low
tolerance for buggy applications. In a never-ending attempt to appease their user base, developers must therefore test their
products across:

 Handset Maker/Model
Did You Know?
 Operating System
“According to mobile advertising startup AdMob, there
 Browser are some $200 million worth of applications sold in
Apple’s iPhone store every month, or about $2.4
 Wireless Carrier billion a year.”

 Location How Big Is The Apple iPhone Economy?

- Om Malik, GigaOm
 Language

No matter what type of mobile applications you develop – whether they’re chat tools, social networking, games, business apps
or others – you face the same fundamental problems encountered by all mobile app developers.

Let’s now take a quick look at how mobile app developers have attempted to deal with these problems…

A Free eBook from uTest Page 2

In-House Testing
Why it could work: In an ideal world, mobile testing could be performed under
the watchful eye of product managers, developers and QA managers, who could
observe firsthand how the application performs under every possible user Just how important is global
scenario. The assembled testing team – consisting of perhaps a dozen or so of the testing coverage?
most talented testers on the market – would have hundreds of handset models at
their disposal: iPhone, Blackberry, Nokia, Samsung, Sony - you name it, they test
“By the end of this year, the mobile
it. It would be a testing lab that even the largest software company would envy.
ad market is expected to grow to
With this type of control over the testing process, what could possibly go wrong? $913.5 million. And by 2013,
Gartner expects mobile ad spending
Why it won’t work: The truth is, for mobile apps, building a comprehensive in-
to surpass $13 billion, with the
house testing lab is prohibitively time-consuming and expensive. This is true for
almost every mobile app company, with very few exceptions. Asia-Pacific region bringing in the
most revenue, followed by North
Imagine the expense of building an in-house team and lab capable of assuring the America and Europe.”
functionality for iPhone, Blackberry and Android handsets (of all makes and
models) across wireless carriers in the U.S., U.K, Spain and France. Now imagine
- Meghan Keane,
this with 3 apps… 5 apps… Will you send testers around the globe to assure
quality? Of course not. Thus, it’s no surprise that mobile app companies – from
start-ups to large enterprises – don’t rely solely on an in-house testing resources.

To be fair, most companies never intend to create an in-house team like the one
described above. Worse, they spend their money trying to get “what they can”
from the method. That is, they have 3-5 in-house testers, with a few dozen
devices (at most). But under this plan, their testing coverage is confined to their
own office, and the whole point of comprehensive QA is lost.

A Free eBook from uTest Page 3

Outsourced (Offshore) Testing
Why it could work: Of course, when mobile application developers discovered
that the in-house route couldn’t provide the testing they needed, they logically
said to themselves: “I’ll just offshore it.” This was not an isolated decision. It Common User Complaints:
was made by thousands of software companies, and few could fault them for
their logic at the time. If comprehensive mobile testing was to be achieved, •Failed or stalled downloads
then surely an offshore firm (with their low-cost labor) would be the only entity •Credit card verification delays
capable of getting the job done at a reasonable price. •No screenshots
It’s this line of thinking that continues to define the way in which many mobile
•App performance when there is
companies deal with their testing objectives. slow connectivity
•Inconsistent buttons, fonts, etc.
Why it won’t work: While outsourcing has certainly proven to be less •Missing or broken links
expensive than hiring an internal team, developers who contracted with such •App not supported by device
firms quickly found themselves outside of the testing loop – losing what little •Outdated versions; bugs still in app
control they had over the QA process. And for what? With no insight into the
day-to-day testing activities, how could these developers expect to improve
their application in a timely manner? Developers must remain close to their
application at all times, which is antithetical to the outsourcing mentality.

And as it turns out, even the largest of traditional outsourcing firms don’t have
the resources needed to provide adequate testing coverage required by today’s
mobile environment. Just like their clients, they cannot easily (or affordably)
test an app across dozens of handset makers, models, carriers, languages and
locations. Plus, the testers employed by these offshore firms are unlikely to be
among the target audience for a company’s mobile apps.

A Free eBook from uTest Page 4

Why it could work: Frustrated by the shortcomings of outsourced and in-house
testing, some mobile app developers decided to turn to technology. That is,
they came to view the use of emulators and simulators as the easiest way to Mobile App Purgatory
improve their testing coverage. In terms of usability and design, these tools
demonstrated their value rather quickly, as developers were able to easily Bugs in your mobile app cause
assess the basic aspects of their application (data input, screen size, button unintended consequences. Take
use, etc.) from the convenience of their own laptop. this story for example: A mobile
app developer is alerted to a
This, it was thought, would eliminate the bulk of mobile app bugs. It didn’t. iPhone bug, and instead of being
able to fix it right away, he spends
Why it won’t work: One of the biggest challenges for mobile developers is that
the testing of applications is occurring in an environment far removed from the weeks pleading with Apple to
real world, where actual users run and interact with those applications ON their allow him to proceed with the
devices. Said differently, the gap between “in-the-lab” simulation and “in-the- update. If bugs in your app
wild” usage is far too great to ignore. appear, getting them fixed quickly
can be very problematic.
Consider, for instance, an application that is data entry intensive. To assume
that it can be fully tested on a simulator (i.e. with full keyboard and mouse
access) and be ready for release is a mistake mobile app developers have Of course, if the bug had been
continually made for the past five years. The convenience of simulators and discovered BEFORE launch, this
emulators has made it easier than ever to be lured into a false sense of would be a non-issue.
security. But the advantages of such tools are limited in scope, and should
never be considered a substitute for real-world, on-device testing.

A Free eBook from uTest Page 5

Beta Programs
Why it could work: For a software company, what could be better than sitting
back and having hundreds, perhaps thousands, of users eagerly providing you
with around-the-clock bug reporting and product feedback - for free?
The Future of Mobile
Needless to say, the allure of an active, engaged group of beta users is too “As mobile matures, more
great for many developers to pass up. Not only do they gauge your application’s
development tools and standards
performance before the big release, they act as advocates for your product,
adopting a strong sense of ownership and helping you to establish a customer will emerge. And in time, testing
base for your product. And so if you’re at all uncomfortable with your app’s tools and bulletproof practices will
performance prior to launch - and don’t want to go to great lengths running evolve. Until then, building and
professional usability tests – beta programs seem like the easiest, safest route. testing mobile apps for the iPhone,
Blackberry, Android, Palm Pre and
Why it won’t: At least that’s how it’s supposed to go in theory. In practice, it is
others will continue to feel a bit like
rare for a software company to attract such a large group of beta testers. After
all, not every company can be Google, with its various (and wildly popular) beta the wild west.”
versions, where people line up to test drive their software. Chances are, your
-Matt Johnston
app either lacks enough mass appeal, or is relatively unknown (for now), and is
hence not that attractive to the beta testing types. VP of Marketing, uTest

But even if you are able to assemble a large beta group, the method still falls
short of providing adequate testing coverage. And the reason for this is simple:
beta testers are more often like users, in that they will try to get your app to
function properly, whereas a real tester would attempt to “break” your app.

In other words: Users use. Testers test.

A Free eBook from uTest Page 6

Crowdsourced Mobile Testing
Combining the benefits of the aforementioned methods, the use of
community-based testing – or crowdsourcing – has altered the way companies
test their mobile applications. This shift has empowered progressive mobile What is Crowdsourcing?
app companies to make the best decisions, on a per-project basis, without
sacrificing control, costs or accountability. Coined by author Jeff Howe, "Crowdsourcing is the act of
crowdsourcing - if you are unfamiliar with the term - is simply defined as: taking a job traditionally
performed by a designated agent
“…the act of taking a job traditionally performed by a designated agent
(usually an employee) and sourcing it to a large group of people in the form
(usually an employee) and
of an open call.” outsourcing it to a large group of
people in the form of an open call.”
By leveraging a global community of professional testers, QA teams are doing
the impossible: maintaining app quality, achieving broad testing coverage, - Jeff Howe
meeting launch dates and staying within tighter budgets. Author of Crowdsourcing

While the thought of having a virtual QA team from around the globe might
seem like a complicated free-for-all, the reality is just the opposite. With the
right online platform, the process fits neatly into your existing teams and
processes. Here’s a quick look at how it works:

1. A testing manager (TM) specifies his QA requirements (by location,

language, device, carrier, etc.) and builds a virtual testing team

2. The TM then uploads his test plans, use cases, known bugs and other

A Free eBook from uTest Page 7

Crowdsourced Testing (continued…)
3. The selected members of the testing team are invited to the project

4. Bugs are reported and classified by the testing community in real-time – On Crowdsourcing
along with documentation like screenshots, video capture – and monitored by
the TM (who can approve, reject or request more information from testers) “Peer production is more than just
sitting down and having a nice
5. The company pays only for the test cycles it needs (no downtime or long- conversation. It’s about harnessing
term commitments). Testers are paid for approved bugs, completed test scripts a new model of production to take
or surveys.
innovation and wealth creation to
The net effect is that mobile app companies get instant access to a global new levels.”
community of testers via a platform that enables detailed management of the - Eric Schmidt
entire test cycle (or even connects to their in-house bug tracking system). CEO, Google

This community can serve as a virtual testing team (for small companies and
start-ups), or as an on-demand extension for larger, more established in-house
QA departments.

A Perfect Match: Crowdsourcing and Mobile App Testing

“No other industry is so uniquely suited to the principles of crowdsourcing
quite as mobile application testing,” said Doron Reuveni, CEO of uTest. “When
your users are dispersed around the globe, your testers should be too. Now,
any company that produces mobile apps can get the benefits of having world-
class QA coverage.”

A Free eBook from uTest Page 8

Crowdsourcing Challenges: Quality and Control
Crowdsourcing has a reputation for being noisy and chaotic, without much actual
productive work. Thus’ it’s critical to choose a reputable community that can Mobile Challenges
point to its customers’ successes, and shares the past performance and
reputation of community members. In short, make sure you’re choosing a “Mobile applications come with
community and not a mob. Other common challenges include: their own unique set of challenges.
Communication: As with any successful project, communication is key. While
Between Wi-Fi, 3G, Edge and all
crowdsourcing doesn't enable face-to-face conversations, it does enable decision- their different behaviors, there are
makers to converse with their virtual QA team in real-time through an easy-to-use so many angles you have to cover.
online platform. Also, make sure the testing company you choose provides a Until recently, testing for this
dedicated project manager to help you manage the community and the process. criteria would have been
Intellectual Property: IP protection is sometimes a concern for customers who
impossible, but crowdsourcing has
are new to crowdsourcing. While it might seem like your IP could be exposed to a changed all that.”
community of thousands, top crowdsourcing companies allow customers to
carefully select their testers and establish non-disclosure agreements (NDA) with -Travis McElfresh,
their community members. -VP of Technology, 1Cast
Quality: With crowdsourcing, it’s easy to believe that anyone can test your
application. Fortunately, the crowdsourcing model enables customers to be as
selective as they want – in terms of testers’ skill sets, experience, performance
ratings – in determining who gets invited to test their app.

As with other methods, unexpected problems do arise, but what makes

crowdsourcing different is its inherent ability to adapt to changes on-the-fly.

A Free eBook from uTest Page 9

Benefits of Crowdsourced Mobile App Testing
Although the advantages of such an approach are numerous, here’s a quick look
at what to expect when implementing crowdsourced QA for your mobile app:
Why It Works
Cost Control: Crowdsourcing allows managers to utilize lower-cost outside
support without being tied down by long-term commitments. The fundamental “Because crowdsourcing is built
structure of online communities creates competition, making it more from the open source template,
accountable, efficient and cost-effective than traditional outsourcing. any company looking to leverage
computer professionals has the
Dealing with Increased Complexity: Significant quality improvement is achieved
when development and testing is done across a wider set of handsets, carriers,
advantage of an audience already
locations and languages. Crowdsourced mobile app testers can be easily recruited familiar with the basic idea of
based on a wide variety of criteria. community production.”

Faster Time-to-Market: Crowdsourcing enables you to ramp up or down to meet -Jeff Howe
your needs at any given time - eliminating the delays and staffing constraints that
often stifle mobile app companies around peak release times.

Higher Quality: Customers expect their mobile apps to be fully functional and
bug-free from day one. Companies that believe “good enough” is good enough
when it comes to quality and usability will learn a valuable lesson (the hard way).

Retaining Users: If your product has problems, or doesn’t meet expectations, it’s
easier than ever for your customers (and prospective customers) to find your

A Free eBook from uTest Page 10

Testing no longer has to be a neglected phase of mobile application Who’s Crowdsourcing?
development. There was a time when the testing matrix was indeed too The number of mobile app companies using
complex, too burdensome to be completed using standard means. But thanks crowdsourced testing is growing everyday.
to the rapid evolution of crowdsourced testing, all that has now changed. Here’s a few companies ahead of the curve:

The companies that are now leveraging crowdsourced testing - particularly in

the mobile space - are gaining more of a competitive advantage than they
perhaps even realize. As the mobile market doubles and triples in size over the
next decade, those that have made testing coverage a priority will enjoy their
ROI in terms of increased market share, profitability and above all, user loyalty.
Those who continue to neglect the testing phase will struggle at best, and at
worst, cease to exist. It’s that simple.

Either way, the future of mobile applications remains bright. Once reserved for
a tech-savvy niche in the business world, the use of mobile applications is now
firmly entrenched in the mainstream. There’s no turning back when it comes to
mobile applications, and that includes methods for testing them.

For more on how crowdsourcing can complete your mobile testing efforts,
chat with one of our QA gurus by clicking here or by calling 800.445.3914.

A Free eBook from uTest Page 11