Papers presented at the ICES-III, June 18-21, 2007, Montreal, Quebec, Canada

Interface Design and Testing for Electronic Self-administered Survey Forms using Excel
Emma Farrell, Kettie Hewett, Tracey Rowley, Leone Wardman and Rob Burnside Australian Bureau of Statistics. increase commitment to the later standards. An ABS Quarterly Economy Wide survey form was converted from paper to Excel as the basis of the tests. This is a long-running survey collecting standard accounting items including detailed income and expenses information. The form was designed to look very similar to the standard paper form with a few exceptions, including: The coloured background was done in several shades of grey only, rather than the colours used in printing the paper forms, because the representation of colour on respondent monitors may be inconsistent. The font used for text was Arial (rather than Times) as this font is good in terms of readability and scalability on screen and also looks reasonable when printed. Instructions were added to explain how to navigate, save and submit the form, as well as indicating that only the white answer fields could be edited. 2.1 First Round: Testing Navigation The first round of testing primarily examined navigation, using four versions of the form each with a different method of moving through the worksheets. The different versions were: Tabs: The standard Excel worksheets were used with tabs at the bottom of the screen for navigating between worksheets. Each worksheet represented a page of the paper form and the label on each tab referred to that page i.e. Page 1, Page 2. Scroll: This used a single worksheet for the whole form, so the user had to scroll all the way through. Buttons: Three button devices common to electronic form interfaces were used with "Prev Page", "Next page" and "Front Page" on them, presented at the top of the screen. Links: Hyperlinks with the standard underlined blue text were used, presented at the top of the screen. Initially there were three hyperlinks with the same words as the buttons above but this was considered too similar. During the tests these were changed to match

Abstract The Australian Bureau of Statistics (ABS) has been using Excel to design self-administered business forms for several years. To minimise mode effects, the original design was very similar in appearance to the corresponding paper forms and had limited functionality. This paper will describe some of the form design principles the interface was based on, and the development and testing which guided the evolution of our Excel form design standards. It will highlight where the Excel forms remain similar to the paper versions, where they are different, and why. Excel forms are now used by several ABS business surveys and are quite popular with certain types of respondents. Keywords: Electronic forms, Self-administered, Screen design, Navigation, Usability testing. 1. Introduction The Australian Electronic Transactions Act (ETA) 1999 requires the ABS and other Government departments to accommodate businesses and individuals who insist or need to report electronically. For ABS business surveys, the Microsoft spreadsheet product Excel was considered the most cost effective and viable option in the short-term. It was intended to be used as the 'ETA fallback' electronic data collection vehicle while development work continued other electronic forms. Excel forms were already in use for some ABS surveys, however a standard design had not yet been developed. Resource issues and the continued popularity of the Excel forms with respondents have led to this method remaining the ABS's primary electronic data collection method. 2. The Original Usability Tests The ETA fall-back Excel forms were intended to be as cheap and easy to produce as possible, but usability testing was still required to ensure the design would be reasonably acceptable to respondents. Testing was conducted on ABS premises using volunteer staff as test subjects. This was cost-effective and also helped to raise awareness of the research within the ABS and

443

Papers presented at the ICES-III, June 18-21, 2007, Montreal, Quebec, Canada

the tabs instead i.e. separate hyperlinks referring to each page, but were kept at the top of the screen. In order to determine which navigation device was more favoured by users, each version was completed by a separate group of 6 test users, making 24 users in total. Each user filled in a debriefing questionnaire about that version, and then was briefly shown a different version so that each version was presented to 12 users. The order was varied so that each possible combination and order appeared twice in the 24 tests. This allowed a points system to be applied: every time a version was preferred over the other, it received 2 points. A version not preferred received 0 points, and each version received 1 point when neither was preferred. It was expected that the points system would show a clear pattern by itself but this was not the case. The total points for each version were too close to discard any version on points alone so the stated reasons for the user's preferences, and related behaviour and comments, were examined. 2.2 Navigation Results The scroll version did much better than expected. The points allocated for users' preferences came out too close to be significantly different, but the scroll version obtained 13 points (out of a possible 24). The users preferred this version's way of moving through and finding the data entry fields, generally using tabbing, arrowing and page down. The tabs version did not do as well as expected. This was clearly due in part to the tab labels not being informative enough, which can be addressed easily, as well as a bias among some of our users due to experience with another spreadsheet application which will be explained later. The tabs version came last in the preference scores with 10 points. The buttons version was the only one to use macros, to create the button navigation devices. When the form is opened, a standard message comes up asking users if they want to disable or enable the macros. At least 5 users disabled the macros that went with this version right away, explaining that was what they would normally do. One of those users also disabled the macros automatically the second time he opened the form as well. A further user was quite confused and distressed at the disable/enable point and selected more information before enabling. This version would also not navigate to the next page while an open cell is still selected- the cursor focus had to be moved to somewhere else on the form. All six

users who tried this version first had this problem. Several found it quite annoying and one never figured out what was going wrong. For these reasons the button version was not used later. However the preferences are still valid and the button version got 13 points. The hyperlinks version was changed twice during the rounds of testing: once to increase the number of links to be more similar to the tabs and less similar to the buttons, and once to make the selected link paler than the others to help the user know which page they were on. These changes seemed to polarise user's opinions of the version, rather than sway them all in one direction. The hyperlinks version obtained 12 points. Overall none of the methods emerged as more strongly preferred by users than the others. The main test results relating to navigation were: labels on spreadsheet tabs, links or buttons that refer to page numbers in the form weren't very useful; using macros to solve any of our design problems was out of the question; the form needed to provide much better keyboard navigation; and the varying positions of the navigational devices was a confounding factor in assessing their functionality. There were clear patterns in the preferences given by the users in this test. Expectations of where the navigation devices should be based on past experience had a strong influence on which method users are most comfortable with. Users with extensive Excel experience tended to prefer the Tabs over Links or Buttons, and users with more experience of a different spreadsheet application used in the ABS, Lotus 1-2-3 (which has tabs at the top), preferred Links or Buttons over Tabs. As the intended audience of these forms were Excel users, navigation devices at the bottom of the screen would be the most effective. However, the tabs version needed meaningful labels for the tabs based on the section headings ("income", "expenses" etc), to satisfy the users wanting to go directly to specific data items, and those wanting a table of contents. The length of form content for each tab would be longer than a single screen in many cases. This was not considered a problem based on the unexpectedly large number of users who preferred the Scroll form. The test users did actually read the "How to use" instructions, because they complained when the instructions were not accurate. If related questions were kept together under single tabs, and instructions were provided on how to move and search across

444

Papers presented at the ICES-III, June 18-21, 2007, Montreal, Quebec, Canada

worksheets using the keyboard rather than clicking on tabs, this kind of user should also be satisfied. 2.3 Potential Mode Effects Using an almost exact replica of an existing paper form on screen also provided an opportunity to witness some respondent behaviour that could reasonably be assumed to have occurred due to the form being electronic. 2.3.1 Independence of pages When a respondent has a paper form in front of them, even though they focus on one page at a time the entire form is still visible and tangible. Moving back and forth in the electronic version is more difficult, although appropriately labelled tabs allow easier searching for items when the respondent doesn't know where they are. The paper form also has continuity between questions due to facing pages, because the previous and future questions are still in the peripheral vision. Many ABS forms extend matrices and question sub-sets across the two pages. This is obviously not possible in an electronic form. Respondents are subsequently more likely to consider the "page" on their screen more as an independent whole than a respondent with a paper form would do. Possible evidence of the problems with this perspective from the testing includes the following: Users were quite confused by the worksheet that only contained notes. They scrolled all over the screen looking for somewhere to enter data, seeming less able to connect that worksheet with the following one than you would expect from a paper form, especially as this part would be page 2 of a booklet facing a page 3 with data entry fields on it. Part of the problem was that the users who happily used the tab key through the rest of the form could not use this to go down the notes worksheet. Users did not understand what the first question was about, because the question itself was on the worksheet of notes but the data entry boxes were on the next worksheet. That worksheet began with "Part 1 Income items continued", but without an instruction indicating "Question 1- Sales income continued", just as the paper form has. Several users were quite confused as to what to include in those answer boxes, one stating that there wasn't a heading for that question. More frequent and appropriate headings and sub-headings seemed important in electronic forms.

Users were unsure if the instructions at the beginning of the form applied to the later sections. Several users grouped the general instructions with the first section's instructions as they were all on the one worksheet. It seemed important to repeat instructions more often within electronic forms than paper ones. 2.3.2 Correction of errors Correcting mistakes on paper forms is troublesome, especially when respondents need to fit each digit of their answer into its own box for scanning purposes. It seems reasonable to assume that the ease with which respondents can change their entries in electronic forms will improve the data quality. The relevant evidence from the testing includes the following: Ten out of 24 users filled out the first question in whole dollars rather than thousands as instructed. The size of this problem was concerning as it was larger than the general size of this problem for ABS paper forms (up to 20% of respondents depending on industry and around 6% for this survey). Interestingly, 8 people realised they had done it incorrectly and all of them went back and changed their answers. It seems unlikely that so many respondents would fix their mistakes on the paper form. One user also cut and pasted the answer to an openended question because it turned out to be in the wrong place. The burden of rewriting what could be several sentences of text was removed, which should lead to an improvement in quality and a reduction in editing along the lines of dealing with crossing-out, liquid paper, arrows going from the box where the text was to where it should be, and various other means respondents might use to repair their mistake. 2.3.3 Expectations for electronic forms Faced with an electronic form, people may expect extra functionality compared to a paper version and get annoyed when it is not present, increasing their cognitive burden and reducing the attention they give the content of the form. For example: Users had a very strong expectation of automatically calculating totals. Some users also expected an edit and subsequent error message if they entered an incorrect total. One user said he would be "very annoyed" if he received a form like that one and it didn't calculate totals. Another user refused to calculate the totals manually and left the boxes blank. One user expected that on-line help would be available, despite knowing that respondents can still

445

Papers presented at the ICES-III, June 18-21, 2007, Montreal, Quebec, Canada

phone the ABS and contact information is always included on the front page. One user mentioned linking to other spreadsheets so that values could be imported into the form, especially for longer surveys. Users attempting to do this and failing could be a disaster, and there may be data quality problems associated with uncontrolled cutting and pasting individual numbers across that don't actually match the definitions of the questions. This happens with paper forms too, but the electronic form may make this much easier and therefore more frequent. Some of the standard layout features of our paper forms did not work on screen, as the expectations of users led them to interpret the features differently to what was intended. Using thicker borders for our total boxes in the same way paper forms do was a major problem as many of our users saw the thickness as indicating the cell was selected, even though it did not actually look the same as a selected cell. Users would begin typing and not understand why the text would not appear in the box or why error messages came up. 2.4 Second Round: Testing Improvements After combining the best navigation elements into a single form, and addressing other design problems, testing was done again to see whether the changes were actually improvements. A sample of five people from the previous test and four new people were selected. A different scenario was used, with additional tasks requiring users to change some of the data. Changes to the form included the following. Spreadsheet tabs were labelled with sections of the form. Automatic totals were included for some questions. Instructions about keyboard navigation were added. The general instructions for the form were placed into their own worksheet, separate from the instructions for the first section. Finally, notes on the screens to indicate where "No data entered in this page" and where the "End of.." each section were used to prevent users scrolling too far. 2.5 Summary of Results 2.5.1 Navigation through the form The offer of a combination of navigation methods, across as well as within worksheets, seems to have been quite satisfactory with a few exceptions. All the users rated the navigation as easy or very easy, with mouse preferred over using the keyboard. Some results were as follows:

Labelled section tabs for each worksheet were very well received and greatly assisted in navigation through the form. Using the tab key between fields was mentioned in the instructions, which encouraged users to apply this method. However, the results were mixed in part because this caused users to skip past the instructions and automatic totals. The addition of specific instructions on other keyboard use for the form, such as arrow keys and Ctrl Page up and down, clearly increased keyboard navigation. This was important from both a user preference and a health and safety perspective. The note added to the worksheets only containing instructions which said "No data entered in this page" was unclear. One user realised that this was intended to mean "No data to be entered in this page" but noted that it could be interpreted as an edit message meaning "No data entered in this page yet". This standard was later changed to "No data to be entered in this page". 2.5.2 Entering and changing different data Rounding figures to dollar thousands: Problems with respondents entering whole dollars, as demonstrated in the first test, appeared to be reduced by the change in layout which placed the general instruction box in a worksheet by itself preceding the first section's own notes and questions. Seven of the nine users, including all four new users, entered dollar thousands correctly right from the beginning. One user wanted "accounting" formatting in the answer boxes so that commas were inserted automatically and this function was later added. Changing the business name: The business name was pre-entered in the Excel form, partly to be more like a posted mail form. This test included a business name change, to see how the users would report this. The users had some difficulty selecting the text and no users wanted to strike-through the name or made any comments in the form to indicate that the name had been changed, which was a concern. Changing numerical answers: Users were required to reopen the form and change some of the values, increasing one value and swapping two others. In most cases users were able to do this very easily, with a few exceptions including one user who, when swapping the two figures, cut and pasted the whole cell rather than the data in it. This ruined the total formula for that question and also removed the border of the answer box.

446

Papers presented at the ICES-III, June 18-21, 2007, Montreal, Quebec, Canada

Automatic totals: These were received very positively although some users were confused by the slight difference caused by rounding and other users assumed that larger differences, due to data entry errors, were caused by rounding. 3. The Excel Form Standards The first ABS standards for all ABS business survey Excel forms were released in early 2002. In addition to all the issues above, these standards included things like ensuring the forms have appropriate page breaks so they can be printed easily and emphasising the importance of pre-despatch checks to ensure the focus was in the right place for each worksheet when opened. The standards also incorporated example templates with all the standard worksheets to make it easier for developers to do things correctly from the start. 3.1 Implementation for Existing Electronic Reporters One of the first pilot tests for Excel forms developed according to the new standards was introduced for the September 2002 quarter of the Local Government Finance Statistics collection. This survey already used Excel forms for the majority of the respondents but the original design of these forms were quite simple spreadsheets which varied in content across states. The changes resulted in significant improvements in response rates, timeliness, and data quality. A post enumeration survey (PES) of 22 respondents was conducted in early 2003. The PES involved visits to the respondents using a detailed interview script to discuss issues with the new form. Local councils appeared relatively homogenous, technically capable in terms of being very familiar with Excel, and amenable to a specialised collection approach as a distinct respondent group. One aim was to investigate whether it would be valuable to expand the existing secure electronic reporting to include a range of data for different ABS surveys. Combining data requests would minimise respondent load through reducing points of contact and number of individual collection instruments ABS despatches. The Excel form and the web-based lodgement procedures were both very well received. The respondents were in favour of electronic reporting generally, however the financial areas which completed the current electronic survey form were not involved in a wider reporting role. One reason for this was the inability to separate out Excel worksheets for distribution to other areas in the council, such as

environmental and human resources (specifically payroll) departments. Individual electronic instruments would still have to be despatched to each council area of interest to the ABS. Suggestions for improvement to the Excel form from the respondents included: provide further text areas to each worksheet that would aid in revisions to data and encourage comments pertinent to each section, include an electronic version of comprehensive notes and classifications, give a unique receipt number along with an acknowledgment that data has been lodged, and provide 'year to date' running automatic totals to aid quarterly reporting. The first two changes were implemented and the third is planned. This survey now has the whole sample responding using the standard Excel forms. 3.2 Spread of Standard Excel forms throughout ABS business surveys There are currently about 40 ABS business surveys using Excel forms every year to a certain extent. The respondents reporting via these forms include large businesses in general economic surveys, government organisations, and Internet Service Providers (ISPs). The standards have been updated regularly to address new requirements and changes in the technology. The Excel forms remain quite expensive to produce and process so the ABS does not explicitly offer this mode on a general basis, at despatch or during follow-up. Respondents already reporting electronically using less standard methods are encouraged to use the standard forms. Respondents who are reporting by paper or telephone and request to report electronically are considered on a case by case basis by a central group. The ABS International Investment collection has the most advanced Excel forms in terms of functionality. These forms are used to collect data from about 8% of total respondents, who are mainly the very significant contributors to estimates such as banks. These forms have complex matrices and some items with a set but large number of possible answers, including countries of investors. Drop-down pick lists were added for these items. Some respondents in this survey also need additional space to report some items and it would be impractical to make each response space large enough to accommodate everyone. These Excel forms therefore have the capacity (via a macro button) for the respondents to 'add rows' to questions to allow enough space.

447

Papers presented at the ICES-III, June 18-21, 2007, Montreal, Quebec, Canada

A limited number of the respondents have also been given the ability to tailor their forms by adding rows once and then use this form as a master or template they copy and resubmit every quarter. The date references (in every question) can be changed automatically using an additional worksheet at the end called a metadata sheet. Feedback from respondents during field testing for this survey in 2006 was positive. Respondents were willing to cut and paste data from other electronic sources and even change the format of the other sources (e.g. to thousands rather than whole dollars) to better fit with our requirements. However, further automation was desirable. These design complexities extend far beyond the original plans for use of Excel as a simple way to represent our paper forms electronically. This complexity has nevertheless worked due to limiting use of the forms to on-going respondents with whom the ABS has a quite close relationship, in addition to the use of standards and testing. 4. The Electronic Form Audit Beginning in July 2006 an audit of electronic forms used for business survey respondents was conducted. Although ABS policy requires all ABS business forms to use the Forms Design Standards and most form development is now centralised, survey areas still sometimes develop special arrangements with respondents to encourage full response. Some respondents also insist on reporting electronically using spreadsheets and other formats they have generated themselves. The purpose of the audit was to understand the current situation in order to manage the growth and nature of e-reporting to the ABS in the future. The audit identified instruments that were not currently to ABS forms standard, or did not maintain conceptual consistency with the paper forms. It also identified inscope instruments that needed to become part of the formal approval process or that were not following other ABS policy on data capture e.g. in terms of security. Extra corporate funding was provided to remediate these forms. A key finding from the audit was that there were some quite extreme departures from ABS Excel form standards to meet the requirements of the surveys and reduce the burden of the respondent. The main example was a number of Excel forms which presented each survey question as a column heading, rather than

appearing down the left side in the same way as the paper form equivalent. The questions were often summarised to fit, and thus did not contain standard question wording and did not comply with many of the regular Excel form standards. However, these forms were particularly suitable for surveys that required multiple responses for different business units on the same form. Each row in the spreadsheet potentially contained the response for an individual business unit, at a level which might be specified by the ABS or chosen by the respondent for ease of reporting e.g. information from different states in Australia, or different departments in a university. Some visits to respondents using these types of forms confirmed their usefulness but also emphasised the importance of including all the necessary instructions and definitions electronically along with the form. For these specific cases exemption from the existing standards was granted and new standards are being developed. These include requirements that these different Excel forms: contain the standard components for all forms such as a "front" page (covering the purpose of the survey, and statements about confidentiality, help available etc.), use the Excel form standard fonts and colours, have the column headings containing the questions frozen so that they remain visible when scrolling down the rows where data is entered, and are accompanied by a PDF of the complete form with all instructions. 5. Future Developments At present most of the Excel forms used by the ABS must be manually processed. In order to automatically extract the data provided, some further work is necessary to determine the most appropriate method to store the systems metadata associated with each potential response. In the future the ABS will use named references for answer fields in our Excel forms. Names can be for single cells, adjacent cells, or the same cells over multiple sheets. For example you could name the entire range of cells which belong to the income questions "tot_inc", while the individual cells within "tot_inc" will have their own names. The names given to cells should be the same as the metadata names used for fields in corresponding instruments for editing and other procedures, so that paper form data and electronic form data will align.

448

Papers presented at the ICES-III, June 18-21, 2007, Montreal, Quebec, Canada

6. Conclusion What began as an interim tool chosen for its relatively low cost has become a successful electronic reporting option. The ABS expects to continue using Excel forms long into the future for suitable surveys and respondents. These will be used at the same time as other electronic options currently under consideration, employing email and web technologies, as part of a wider multi-modal data collection framework. Acknowledgements Thanks to Greg Griffiths, of the Australian Bureau of Statistics, for presenting this paper on our behalf at the conference.

449