You are on page 1of 14

Best practices in mobile research

Manuel Zahariev, Chris Ferneyhough, Chris Ryan ESOMAR Online Research, Chicago, October 2009

Title: Author(s): Source: Issue:

Best practices in mobile research Manuel Zahariev, Chris Ferneyhough, Chris Ryan ESOMAR Online Research, Chicago, October 2009

Best practices in mobile research

Manuel Zahariev, Chris Ferneyhough and Chris Ryan Vision Critical, Canada Sonia Bishop Angus Reid Forum INTRODUCTION For the past few years, online data collection for marketing research has been established as a robust, mainstream methodology. Until very recently, data collection using mobile technology was primarily confined to SMS surveys and intercept-based research. SMS-based surveys consist of a series of exchanges between a survey system and the respondent, where the system sends a question in an SMS message and the respondent types their choice in a message. A typical exchange is illustrated below: System: Does our mobile web site provide you with the information you were looking for? REPLY Y FOR Yes OR N FOR No Respondent: Y The system continues the survey by sending the respondent another question. SMS surveys have low resilience to respondent errors and are applicable to very short surveys. Mobile devices are also used to augment traditional intercept survey scenarios. Responses are automatically entered in a database, either through a wireless network connection while the survey is being filled or at a later time, when the device is synchronized. It could be argued that mobile-augmented intercept increases the quality of the responses through the removal of conversion errors and through application-based enforcement of the survey logic. Recent advances in mobile phone technology make it possible to consider the delivery of online surveys directly to mobile devices. A new generation of so-called smartphoneshave web-browsers, email access and the ability to install and run applications. Globally, an increasing number of cellular phone operators offer Internet access; users can browse the web and access email

Downloaded from

on phones with these capabilities. New phones from a number of mainstream manufacturers have the ability to connect to WiFi networks. Wi-Fi points of presence are increasing in coverage and represent, in many cases, faster and less expensive alternatives for accessing the Internet than the access provided by wireless carriers. As a result of these developments, access to the Internet happens increasingly through mobile devices in a number of geographies. Respondents to existing online panels already receive invitations to surveys on their mobile device; in many cases, these invitations link to surveys designed for consumption on a desktop or laptop computer, and on a mobile device can lead to inefficient, painful or completely broken respondent experiences. For example in the screening study for this project, conducted on national panels provided by Vision Critical, a percentage of respondents had received their invitation to a survey on their mobile device, reported by geography United States: 1.9%, United Kingdom: 1.2%, Canada: 3.8%. Obviously, results will vary depending on the composition of the panel. Specifically, our Canadian panel has a higher proportion of higher income and employed members who tend to be more tech-savvy, so the higher proportion in Canada should be interpreted with caution. In this paper, we will be focusing on best practices associated with data collection for web-based online surveys where respondents use mobile devices. DEVICES There are significant differences between the respondent experience in an online computer environment (further referred to as onlinein this paper) and in a mobile environment. The online respondent experience on a computer cannot be directly transferred to mobile devices. For computers, there are a small number of primary, cross-standardized hardware architectures; two operating system families (Windows and Mac OS) cover most of the market; four browsers with comparable and advanced feature sets (Internet Explorer, Firefox, Safari and Google Chrome) cover over 99% of Internet users, with reasonable standardization and comparable, complete feature sets. Generally, online surveys do not push the limits of the computers they are running on. By contrast, there is a wide variety of mobile devices, from more than a dozen major manufacturers, in a variety of models. It is reported that there are over 1,600 different models of smartphones and new models are brought to market frequently. Screen sizes, resolution, processing power, memory, and input modalities vary widely, even between devices of the same manufacturer. Operating systems and application programming environments are also very different. For example, Nokia supports three families of operating systems on their devices, with significant variation even between minor versions of the same family and at least five different programming platforms. Mobile browsers are a lot more variable, with capabilities generally lagging well behind computer-based counterparts. Significant differences are also common in rendering online content, adherence to web standards and support for dynamic page content through client-side scripting. Support for rich content browser plug-ins such as Adobe Flash is very restricted, with significant variation in the supported feature set between devices. Assumptions can safely be made about the existence of a client-side scripting engine (JavaScript), support for cookies, font colours and images in a computer-based browser. These assumptions do not hold on mobile devices; for example, JavaScript is disabled by default on a variety of RIM Blackberry devices; many users disable the automatic image download capability in the browser.
Downloaded from

As a result, providing a uniform and robust respondent experience across a variety of devices, on different browsers, with different display and input capabilities is the most significant challenge of mobile data collection for marketing research. For this study, we used sample from Vision Critical's national panels in the United States, United Kingdom and Canada. These panels are used by clients via omnibus and field & tab services, syndicated studies, public affairs and custom research. Panel members are drawn from across the full breadth of the adult population and profiled against a range of demographic, consumer, social and attitudinal characteristics. Data quality is assured by the application of rigorous panel management practices and recruitment quality standards. Since these panels are operated on Vision Critical's industry online research platform Sparq, they can take advantage of the mobile delivery capabilities of the platform. To assess the situation and identify panellists with smartphones, we conducted a profiling study across three national panels (Canada, United States and United Kingdom). Best practices for profiling include:

Create and maintain a comprehensive list of phone manufacturers; Create and maintain a comprehensive and up to date list of phone models by manufacturer; Identifying if the smartphone is for business or personal use; Determine the email address associated with the smartphone; this can be different than the email address registered with the panel for online access;

Assess interest in the ability to conduct surveys via their smartphone; Determine the panelist's data plans (can influence willingness to do mobile surveys); Determine the panelist's capability and use of internet browsing on smartphone.

In profiling our three national panels in Canada, United States and United Kingdom, we discovered the following differences in the incidence of devices (see Table 1).

Table 1 Seven manufacturers cover more than 10% of panellists with a mobile phone in at least one of the markets; maximum penetration in any country is just over 36% for Nokia in the United Kingdom and 31% for BlackBerry in Canada. The challenge is substantial, since the list of devices for each manufacturer can be quite sizeable; for example, there are more than 30 models of BlackBerry smartphones, and hundreds of Nokia phone models. QUESTION TYPES SUPPORTED
Downloaded from

For this test, we did not support Vision Critical visual questions, grids, or other-specify types. There were two main reasons for this. First, in terms of usability, a longer process of design and testing is required to tailor the question types to the small screen. Mobile devices have their own design constraints and strengths, and simply porting over existing designs or implementations would not result in a good respondent user experience in many cases. Incorporating user input techniques such as multi-touch is ultimately desirable but will take a significant amount of effort for a relatively small portion of respondents currently using an iPhone (see Figure 1).

Figure 1 Second, the implementation environment differs, particularly across platforms. For example, there is no support for Adobe Flash on any of the phone platforms except Nokia, which prevents deployment of most of Vision Critical's visual question types on those platforms. JavaScript is turned off by default on a number of BlackBerry devices, precluding the normal behaviour of other-specify questions (enabling the textbox when the otherbutton is selected and disabling the textbox when any other button is selected). In order to accommodate as wide a range of mobile devices as possible, a lowest common denominator approach to question design was taken. This was still effective, as there are tradeoffs between the increased usability issues for the more sophisticated question designs, and users know how to deal with the standard widget set. The small screen environment requires a dedicated survey implementation. Surveys designed and developed for desktop and laptop browsers will generally work on mobile devices, but the user experience is not optimal. For example, the screen shot in Figure 2 shows a question with a default browser-based skin, displayed in Mobile Safari on the iPod touch (the iPod touch and iPhone share the same user experience). The text is unreadable and the targets are too small to hit.

Downloaded from

Figure 2 The user can read and complete the question by zooming in, but this would be required for every question, and the zooming and scrolling would become laborious and likely result in high drop rates. Similarly, if the question is too long or answers are verbose, the mobile user will have to scroll to see all content and access the Next button. On some devices, it is not obvious that there is additional content that can be accessed by scrolling (see Figure 3).

Figure 3 Grid questions were broken down as single-choice questions, one per screen, and reduced in number: one grid question with six items in an online survey would become six different questions in a mobile survey. Figures 3 and 4 show one of these disassembled grid questions in a desktop browser and in Mobile Safari on iPhone OS.

Figure 4 Open-ended questions (see Figure 5) were used but limited to one or two per survey, to reduce effort in manual text entry using small or on-screen keyboards. Some common-sense guidelines here include:

Figure 5

Limit the amount of text in the question. The more text the less space for your answer categories, and the more scrolling required. It is also important to try to keep the Next button above the fold (see Figure 6).

Downloaded from

Figure 6

If possible, don't state your scale in the question. Edit to remove superfluous words: get to the point. The example in Figure 7 shows how shortening responses makes the difference between the Next button being above the fold or not.

Figure 7

Limit the number of answer categories; each one will take up a line of precious space. It is strongly advised to have no more than ten answer categories.

Limit the amount of text in each individual answer category or wrapping of text will occur, taking up two lines instead of one and thereby reducing the number of answer categories (see example above).

METHODOLOGY Vision Critical conducted a series of surveys to further test the mobile survey space. The main objectives of the surveys were:

Compare online data with mobile data is the data comparable despite the different mode of data collection? If the data is the same, it opens the door for multi-modal studies

Compare results by type of device do different devices with different user experiences affect the data? Compare the experience across different regions are there different issues by region? Use the study to further develop best practices.
Downloaded from

Three countries are included in the research: Canada, United States and United Kingdom. The sample population was smartphone users, not the general population. The demographic profile of smartphone users is different from that of the general population (for example, higher income and more education). Panelists on our Canadian, US and UK access panels were screened to identify the ownership and use of smartphones both in terms of brand and model and whether it was their business or personal phone. Furthermore, to respect our current relationships with panelists, we inquired into their willingness to take part in surveys via their smartphone as we could not presume to contact them in this manner without their consent. The United States and United Kingdom involved a mobile only survey. That is, panelists with smartphones were invited to complete a survey on their mobile phone. They could not complete the survey on their desktop computer. If they attempted the survey via their desktop, they were directed to complete via their mobile phone. We designed the study in this manner in order to have a clean sample of people who completed the survey via a single mode (see Table 2).

Table 2 The Canadian sample of smartphone users was split into two separate samples: 500 smartphone users were directed to a mobile survey and 500 smartphone users were directed to an online survey. The two samples were balanced on gender and age and results were weighted with the same weighting scheme (by gender and age). As the incidence of iPhone users is lower than for Blackberry and other devices, iPhone users were oversampled to have a sufficient number of completes among iPhone users (see Table 3).

Table 3 Based on the research objectives, it was determined that the more important objectives of the research were to examine data comparability between online and mobile surveys. If the data is the same, it opens the door for multi-modal studies. In Canada and the United Kingdom, response rates to the mobile survey among iPhones users are slightly higher than for users of Blackberry devices and other brands. In the United States, response rates for Blackberry and iPhone are similar (see Table 4).

Downloaded from

Table 4 iphone users tend to be: technologically savvy, early adopters, high users of mobile applications, physically and emotionally attached to their phone and willing to use it in every way possible. Their enthusiasm for their phone would partially explain the higher response rates. Data (discussed later in this report) indicates that iPhone users are more likely to have enjoyed the experience of doing the mobile survey and to want to do more surveys on their phone. An additional explanation for iPhone users enjoying the experience more could be that the iPhone does a better job of replicating the desktop Internet experience. The iPhone offers a larger mobile screen than most BlackBerry devices (with the exception of the BlackBerry Storm), and provides the ability to rotate the device into a portrait or landscape view, and provides the ability to easily zoom in and zoom out. While this is no way is an exact replica of the desktop or laptop Internet experience, it is closer to the experience than most BlackBerry devices allow. SURVEY FINDINGS Compare Online Data with Mobile Data As when online became a mode of data collection, many questions have arisen about the comparability of data collected via desktop to that collected via telephone or mail. The same question will arise with mobile surveys. Keeping sample characteristics and survey questions consistent, is the data similar for both online and mobile? The survey topic covered the innocuous but topical subject of the economy. The data revealed remarkable similarities in the results. Both the single choice questions on the Canadian economy and personal finances were almost identical. The multichoice responses were also very similar with no statistically significant differences. In total, six single choice questions and one multi-choice question on the economy were included in the survey. There were no statistically significant differences in results across any of these questions (see Figure 8).

Figure 8: Consistent Result for Online and Mobile Other questions related to:

Downloaded from

Change in the price of consumer goods over the next six months (online vs. Mobile) Up (40% vs. 38%), Stay the same (49% vs. 52%), Down (8% for both);

Change in the economy over the next three months (online vs. Mobile) Improve (39% vs. 41%), Stay the same (51% vs. 48%), Decline (7% vs. 9%);

Canadian economy compared to other economies (online vs. Mobile) Better (76% vs. 78%), Same (21% vs. 18%), Worse (1% for both);

When the Canadian economy will be officially out of recession (online vs. Mobile) 2009 (16% for both), 2010 (53% vs. 54%), 2011 or later (23% vs. 21%).

One open ended question was included related to the economy. What is the best piece of advice you've heard about how to personally adjust to living in a recession? Several analyses were conducted to assess the use of open ended questions in mobile surveys beyond that of obtaining some sound advice on cutting spending, reducing debt and living with your means. The amount of content provided for the online and mobile verbatim question was remarkably similar: the average number of characters was 44 for online vs. 42 for mobile. It is interesting to note some differences by country for the mobile verbatim responses: those in the United Kingdom are similar to Canada at 39 characters while US respondents are less wordy in their responses at 32 characters. Furthermore, the format of the content (phrases, sentence structure) is similar; mobile respondents do not abbreviate their response beyond that which online respondents do. There was also no difference in the proportion of respondents who provided a response of none, naor some other non-response. Compare Results by Type of Device In light of the many different devices and variability in usability across devices, one research objective was to determine if any differences emerged related to device. There were no statistically significant differences in the survey results for questions relating to the economy. While the sample size for other types of devices is small, it is evident, even directionally, that the results are similar the one exception being that iPhone users are less likely than Blackberry users to have refinanced their mortgage (4% vs. 11%)! The Mobile Survey Experience A series of questions were asked near the end of the survey about their experience completing the survey. This was asked of both mobile and online respondents. The vast proportion of respondents across all countries and online and mobile agreed with each of the following statements:

This survey was easy to complete. I enjoyed doing this survey. I found it convenient to do this survey.

Downloaded from


However, the strength of this agreement (strongly agree) does tend to be lower for the mobile sample within Canada. As a note, the US and UK panels are younger panels which are most likely still experiencing a honeymoon period (see Figure 9).

Figure 9: Attitudes Towards the Survey-%Agree The more important factor is whether the option of mobile surveys encourages more participation in the survey process. For all regions investigated, the answer is yes with 36% to 66% stating they would be more likely to participate in future surveys if they had the option of completing via their phone. These results would suggest that the ability to complete a survey on a smartphone is an attractive option for panelists (see Figure 10).

Figure 10: If I Had the Option of Completing Surveys Via My Phone, I Would Be In Canada, iPhone users' enthusiasm for their phone comes through once again as they are the most likely (45%) to take part in more surveys if they had the option of completing via their phone (see Figure 11).

Downloaded from


Figure 11: If I Had the Option of Completing Surveys Via My Phone, I Would Be There were few differences in the demographic profile of smartphone users by device. The small sample size among those with other brands of smartphones made it difficult to detect significant differences, however, there is direction evidence that they are younger and more male. iphone users are significantly younger than Blackberry users (see Figure 12).

Figure 12: Fairly Similar Demo Profiles by Device Canada SURVEY LENGTH The mobile survey requires more time to complete. An analysis of the average time to complete indicated the online version took an average of two minutes compared to four to six minutes for the mobile version. Keeping the questionnaire short is key; all mobile respondents who completed a follow-up online survey indicated the survey was just the right length. CONCLUSIONS As Internet users migrate towards mobile usage of the Internet and decreased usage of the traditional desktop Internet, it will be important for research companies to ensure they have the means and wherewithal to reach respondents on their mobile device or risk losing out on data collection from an important segment of the market. Equally important will be ensuring that the user experience for completing mobile questionnaires is positive to ensure that respondents will continue to complete questionnaires on their mobile device.
Downloaded from


Encouragingly, the research we conducted suggests that when mobile data collection best practices are followed, and when proper sampling techniques are utilized, data collected by mobile devices is consistent with data collected using a traditional online approach. At a minimum, the mobile questionnaire design best practices should include: 1. Design the questionnaire based on the question types supported (single choice, multi-choice, open ends). 2. Design the questionnaire differently (shorter, more concise). Have less text in the questions and answer categories. 3. Limit survey length to 15 questions (although in theory higher incentives can be used for longer surveys). Equally important are the back endbest practices, which should include: 1. Minimize the skin to maximize the screen size/space 2. Profile your panel for Smartphones

Brand and model of device Business or personal phone Email address for Smartphone Interest/willingness to do surveys via phone

3. Identify devices supported in your data collection software

Major brands are Blackberry, iPhone, Nokia, Motorola, Samsung But there are many more and many models

4. Manage deployment process

Process for occasions when email address for mobile surveys is different from main panel email address

5. Determine incentive amount and process 6. Error messages in online studies often rely on JavaScript which can't be done for mobile

Copyright ESOMAR 2009 ESOMAR Eurocenter 2, 11th floor, Barbara Strozzilaan 384, 1083 HN Amsterdam, The Netherlands Tel: +31 20 664 2141, Fax: +31 20 664 2922

Downloaded from


All rights reserved including database rights. This electronic file is for the personal use of authorised users based at the subscribing company's office location. It may not be reproduced, posted on intranets, extranets or the internet, e-mailed, archived or shared electronically either within the purchasers organisation or externally without express written permission from Warc.

Downloaded from