A method of choosing sample devices for field

testing a consumer mobile application

Arya Priya Bhattacharya
Testing Practice, Technology Group

September 2010
Table of Contents
1. Introduction............................................................................ 1
2. Problem Statement ................................................................... 2
3. Solution Proposed ..................................................................... 3
4. Conculsion ............................................................................. 4

For Internal Use Only Page i

Introduction

A phenomenal growth of mobile devices has resulted in mobile applications that deliver complex
functionalities and features. Unlike the desktop based environment, the mobile environment comprises a
plethora of devices with diverse hardware and software configurations. Most applications can be broadly
classified in two categories:

• Browser Based Applications

o These are the applications that can be accessed using browsers on mobile devices.

• Client Based

o These type of applications need to be downloaded and installed on the mobile devices.

From the technical point of view, mobile applications can be differentiated by the run time environment
in which they are executed:

• Native platforms and Operating Systems such as Symbian, Windows Mobile etc.

• Mobile Web/Browser runtimes, such as Webkit, Mozilla/Firefox, Opera Mini, RIM etc.

• Other mangaed platforms and virtual machines, such as Java/J2ME, BREW, Flash Lite, and

Silverlite

This diversity in mobile environments leads to many challenges in application development, quality
assurance, and associated deployment. This document highlights on the approaches that can help in
choosing the devices for testing mobile business applications.

For Internal Use Only Page 1

Problem Statement

The ever expanding penetration of mobiles have resulted into a fierce competition among service
providers to reach the customer. Hundred on new mobile phones and millions of new applications hit
the mobile consumer market every month which makes the deployment strategy of any new
application/service onto a mobile market a substantial challenge. Let us take some examples of
common solutions that providers strive to bring to the consumers:

• Finance: Mobile banking, trading applications

• Travel: Mobile ticketing

• Healthcare: Information access systems

• Entertainment: Games and Utility applications

• Other: Mobile gambling, Advertisement, and other service solutions

With the existing variety of mobile applications, technologies, platforms and devices, development
strategy is genereally more comfortable to arrive at than that of testing. Because the first question to
ask before starting ant test process is "Which devices should we test on?, How do we know if what we
choose will reach the right amount of people?”.

Now this poses significant questions for the decision criterion like, identifying which market segment
will the application best cater to? What kind of phones are the target segment most likely to use?
What are the hot selling phones that must be supported? Do I need to support the devices released in
the past?

For Internal Use Only Page 2

Solution Proposed

There are two major approaches people tend to choose from for sample device spectrum when
attempting to field test a mobile application.
• Device Based
Device based sample selection is generally done by selecting the devices with highest
consumer proliferation as primary test samples (for extensive testing) and other phones in
the same device family are considered as secondary devices and tests on these secondary
devices are a smaller subset of the tests on primary devices
• OS based
OS based sampling however, concentrates on any one device in a OS family as a
primary and the rest of the devices in that family secondary. Any significant changes to
heap memory, screen size, interface would generally mean the OS branching out to
separate build lines

However, the truth is that both are interdependent. A popularity of a device is not enough
reference data to categorize your test samples into primary and secondary. For example, you could
take a popular Blackberry Bold as your primary, and assign a less successful Blackberry Storm a
secondary, that’ll be unacceptable as the interfaces are completely different and your end testing
will not produce effective results. Similarly, choosing an unsuccessful device merely because it
starts the OS line will lead to wastage of resources.

What one should endeavor is to choose the market runners as the primary device sample and then
look at their OS lines for secondary device samples. For the secondary samples, ensure that they
are from the same OS line, it’ll be better to choose mobiles with higher proliferation rate as
secondary if available but focus on devices that are available. It is a frequent hazard, when people
look for an exact device model because of its popularity, where as another model in the same line
will entirely suffice its testing requirements. So what we’re saying is that when considering
secondary device samples don’t shy away from available alternates.

This unified approach also merges the risks of the two, like if the focus is only on primary samples,
medium impact variations of secondary devices cause by interface or input form changes can lead
to gaps in the test environment.

For Internal Use Only Page 3

Conculsion
The best method in our humble opinion is to start by making a list of primary devices based on device
popularity – and make sure you have no more than one device of each OS as a primary sample. Now
that you have a small sample set to work with, build your test plans, test cases and test data against
this set. Once this process in complete build an OMT Checklist (OMT=Operative Matrix Test), what this
means in simple terms is design a certification system of all priority one areas identified while forming
test cases for the initial set (include lower priorities too if really needed). Introduce the list of your
secondary samples to the matrix, and now you have a traceable and measurable sample portfolio in
which to field test your mobile application.

For Internal Use Only Page 4