Usability Evaluation

LBJ Presidential Library Interactive Daily Diary

Prepared for:

R. Bias, Usability Class, Fall 2011 Q. Stewart, IT Specialist

Prepared by: Authors: Date:

University of Texas at Austin, Usability Group Rob Scordino and Hohyon Ryu Nov 28, 2011

2011 Rob Scordino and Hohyon Ryu

Contents Introduction
The LBJ Interactive Daily Diary Design Usability Goals

Purpose of Evaluation
Concerns Usability Evaluation Goals

Target Audience
Background Participants

Design
Outline Logistics Data Collection

Results
Pilot Study Pre-evaluation Questionnaire Usability Tasks Post-evaluation Questionnaire

Discussion Recommendations Conclusion Appendices

A- Consent Form B- Background Questionnaire C- Exploratory Task Protocol D- Specific and Open-ended Task Protocol E- Post-evaluation Survey

16 20 22
24 25 26 27 28

Introduction
This document describes the usability evaluation plan for the Lyndon Baines Johnson Presidential Library Interactive Daily Diary tool. The purpose of a usability evaluation is to predict the expected performance of the actual customer using the current product and materials, as well as detect any serious problems prior to the release of the product. This plan includes the following sections: Purpose of the usability evaluation Target audience Design of the usability evaluation Data collection methodology Forms to be used during evaluation

The LBJ Interactive Daily Diary

The Interactive Daily Diary (IDD) is a tool available through the Lyndon Baines Johnson Presidential Library website. The Interactive Daily Diary highlights several important dates during Johnsons administration. The content of the IDD is made up of digitized copies of LBJs daily diary (the entire diary, from 1959 to 1969 is also available through the LBJ Library website), and document, videos, telephone and other audio recordings, transcripts related to the each of the days highlighted in the Interactive Daily Diary. At its core, the Interactive Daily Diary is a tool for research and education. Potential users for this site could be anyone who is interested in historical documents of the LBJ administration. This means that users will likely have some background knowledge of the topic they are researching. Users will likely be familiar with Internet browsing and research, however they may have never encountered an interface like the one used for the Interactive Daily Diary.

Design Usability Goals

The IDD was designed as an open-ended research tool. Because the Interactive Daily Diary is an openended resource, user goals will vary based on the users needs. The IDD was designed to be content and feature-rich, but it is still unknown how this tool will be used. Thus, the IDD needs exploratory user testing to examine what features will be used and how those features will be used. The evaluation will focus on determining if the needs of the user are met in a easy to understand, useful, and productive manner. Specific measurable goals for the usability evaluation are outlined in the Usability Evaluation Goals section of this document. 3

Purpose of the Usability Evaluation

Until now, a formal evaluation of the Interactive Daily Diary tool had not been completed. The aim of this usability evaluation was to investigate users experiences with the interface and content of the Interactive Daily Diary. The evaluation will attempt to determine if and how users use the features of the Interactive Daily Diary. As such, this evaluation also served as a sort of user analysis and relied heavily upon the use of exploratory tasks and open ended user feedback.

Concerns
Little is known about how this tool will be used. In a discussion with stakeholders of this project, we determined that this lack of knowledge is the key concern that this evaluation should address. Some specific questions to be addressed in this usability evaluation included: 1. Can users find the Interactive Daily Diary tool? 2. Can users find specific items (documents, video, etc) through the pdf interface? i.Are the icons and their meanings obvious? ii.Are users able to navigate between items? 3. Do users use the available tools? 4. How do users modify the UMI (User Modified Interface)? 5. How do users see themselves using the Interactive Daily Diary?

Usability Evaluation Goals

Specific usability goals were determined from the above concerns. These goals allow for the creation of evaluation scenarios and tasks that will let us know if our concerns are valid and what measures can help us determine if in fact the participants are having trouble completing the tasks. This evaluation was based on the following usability goals: Participants will be able to find the Interactive Daily Diary from the main library. Participants will be able to begin using the application with no assistance. Participants will be able to complete activities or locate specific information. Given a scenario, users will be allowed to freely navigate the site to complete a task. Participants will be able to move from item to item.

A survey was also used to determine: Users opinions of icons used for navigation. If and how users would use this tool. 4

Users opinions of ease/difficulty of use. Thoughts on which tools were or were not helpful. User suggestions for improvement.

Target Audience
The selection of participants whose background and abilities are representative of the products intended end user is a crucial element of the evaluation process. Valid results will be obtained only if the participants selected are typical end users of the product, or are matched as closely to the criteria as possible.

Background
The primary purpose of the Interactive Daily Diary is to provide an interface for browsing related digitized primary sources within their context. Thus, this tool is for education-- both formal and informal. Actual target users have not been explicitly determined, but assumptions can be made about potential users. Potential users will be interested in history, specifically historic events from the Johnson administration. Users may want to use this tool for formal education purposes, informal use, or somewhere in between. Users may be any age, but due to the nature of the content and interface, it is likely that this tool will me most suitable for people who are high school aged and older.

Ideal Participants
The following list shows the key characteristics of likely IDD end users. These characteristics are the basis for participant selection for the usability evaluation. The participants will be selected to reflect the range of characteristics shown below. User type Formal educator (secondary / post-secondary) Researcher (undergraduate / graduate) Informal researcher (history enthusiast) Medium High Medium High 5

Computer familiarity

Ability with basic web-based technology (Internet browsing, email)

Educational level Informal education Undergraduate degree Advanced degree

Design
A single usability evaluation was run in five individual participant sessions. The design of the usability evaluations were greatly influenced by Rubin and Chisnells (2008) Handbook of Usability Testing. An evaluation session consisted of (1) participant background questionnaire and orientation (2) performance evaluation of each website, (3) post-evaluation questionnaire, and (4) participant debriefing. Background questionnaire and orientation The participants received a short, verbal introduction and orientation to the evaluation. During this time the moderator explained to them the purpose and objective of the evaluation, and additional information about what is expected of them. Participants were assured that the website was the center of the evaluation and not themselves, and that they should perform in whatever manner is typical and comfortable for them. The participants were told that they will be assigned several tasks, and the moderator will observe and take notes as they complete those tasks. They were also informed that screencasts may be taken during the evaluation. After this was explained, the participants filled out a consent form (Appendix A) as well as a short questionnaire (Appendix B) about relevant aspects of their background. Performance evaluation After the orientation, the participants sat at the evaluation computer. The participant was able to select the browser with which she was most comfortable. The moderator directed the participant to the LBJ Library main site, and the evaluation began. The evaluation tasks consisted of the following:
1. Navigate to the Interactive Daily Diary. 2. Browse site and give initial thoughts 3. Find an image of LBJ and open it. 6

4. Search for LBJs midnight address about the Gulf of Tonkin. 5. Explore the Glifos page and give thoughts and opinions.

Post-evaluation questionnaire After the performance evaluation, participants were asked to fill out a post-evaluation survey. The survey was designed to make the participants evaluate aspects of the site as well determine participants overall satisfaction with each site. Participant debriefing A debriefing session took place after the participant was finished evaluating the Interactive Daily Diary. Any further questions the administrator had following the evaluation were asked during this time. After the debriefing session, the participants were thanked for their efforts, and released.

Logistics
Each participant was tested separately. In each evaluation, the testing environment most closely resembled the simple single-room setup (see figure below) from Rubin and Chisnels Handbook.

Simple single-room setup.

Evaluations were done using both PC and Macintosh platforms. Both pilot tests were performed on MacBooks; the first with Firefox, the second with Safari. Four of the main evaluations were done using a PC; three with Firefox and one with Internet Explorer. The final evaluation was done using a Mac with Firefox.
7

None of the participants required additional supports to meet accessibility needs. In a largerscale usability test, it would be important to involve participants with varying abilities to address accessibility design issues as well. In such cases, modifications would have to be done to the current configuration (such as implementation of screen readers, screen magnification, modified keyboards, etc.) to properly evaluate the usability of these websites.

Data Collection
Data sources: Background questionnaire Moderator notes from observation Participant evaluation Post evaluation questionnaire Quantifiable data collected from the evaluation included:

The amount of time to complete Participants ratings of certain aspects of the website

Qualitative data to be collected during observation included: Subjective opinions of the usability and aesthetics of the product expressed by the participants Indications of frustration or joy from the participant Participants answers to open-ended questions Participants suggestions for improvement

Results
Pilot Test
Through two pilot tests, we could identify practical problems in our usability test design that hamper the evaluation. First, we found that the pre-evaluation questionnaire form should be different from ordinary demographics survey. We asked gender in the form, but it was not necessary as an observer could easily fill in that information. We also asked the age in age brackets. However, the subjects felt uncomfortable to reveal their age in front of the researchers. Therefore, we omitted these fields in the actual test. If the age should be asked, it may need to be asked with a large age brackets. Our subjects had similar ages and digital affinity.

Pre-evaluation Questionnaire
Through the pre-evaluation questionnaire, we collected the basic demographic information of the test participants. We had six graduate students and one teacher. Two of the graduate student subjects participated in the pilot tests. They were all female and they were all Internet experts or had very high Internet affinity. Only one of these seven participants had visited the website before to get general information about the LBJ library, but none of them had used the IDD.

Usability Tasks
Table 1 shows the five participants performance across the four usability tasks. First three tasks were timed and the last one was not times because it was a exploratory task and we wanted collect the users opinions qualitatively.

P1 Task 1. Time to IDD (seconds) Task 2. LBJ Picture Task 3. Midnight address Clicked on the Diary Collection link Yes 45 30 49

P2 75 56 12

P3 32 18 20

P4 25 2 45

P5 40 15 32

Average 43.4 24.2 31.6

Yes

Yes

No

Yes

80%

To click on an IDD entry Found in the index page (Bottom of the IDD PDF)? Clicked on the "KEY"

Success

Fail

Success

Success

Fail

60%

Yes Yes

No No

Yes Yes

Yes Yes

Yes Yes

80% 80%

Table 1. The performance of 5 participants to the usability test (excluding pilots). The participants had hard time finding the IDD from the homepage, and some navigational tools had usability issues.

The first usability task was to find the IDD from the LBJ library website homepage. As shown in Figure 1, it was hidden in the homepage, and even when they know the exact name of the menu, it took 43 seconds to find the IDD. In case of the second participant (P2), it took more than a minute to find the IDD. From the homepage. Also there are other confusing items like LBJs Dairy Collection under Research Collections. It is problematic that LBJs Dairy Collection does not have any explanation how it is different from the IDD, and it also did not have any link to the IDD.

Figure 1. The navigation to the IDD from the LBJ library homepage.

10

Once they come to the IDD page, many people clicked the link to the diary collection show in Figure 2. Since there is no link back to the IDD, people easily got lost and stuck in the diary collection and could never open the IDD entries. Also some participants thought the quick links on the left column was part of the IDD. They clicked on them and used other interfaces to get the tasks done, but they did not realize that the photo archive is not part of the IDD until we explain.

Figure 2. The IDD page on the LBJ Library Homepage. Many participants clicked on the first link (http://www.lbjlibrary.org/collections/daily-diary.html) that brought them to the diary collection. Very few participants clicked on the actual diary entries without explanation. Quick links are also not relevant to the IDD, but people thought they were part of this page.

The second task was looking for any picture of President Lyndon B. Johnson. People could find it easily with the interactive diary interface. However, in the actual diary, people got confused with the KEY section on the left as shown in Figure 3. They are legends that are not clickable, but people tried to click on the icons as they thought it was a navigational tool. The IDD was provided in PDF format, and people found it very confusing as it contains 3 different formats in one PDF file. For example, the second picture in Figure 3 is the index page attached at the bottom of the IDD, and the participants could not easily understand what they were and why they needed to be there. After our explanation or figuring out what 11

the index page was, they liked the idea and used the index page for the most of the tasks as the diary entries are not easily read.

Figure 3. The IDD provided in PDF format. Many people were confused as they thought that the KEY in the left column was a navigational tool. They were also very confused as the format of the index pages at the bottom of the PDF (the second picture) is very different from the first page and they did not expect the new format.

12

The third task was to find the Midnight Address video, and the fourth task was to explore the video interface, which utilized GLIFOS as shown in Figure 4. Finding a video using the IDD was not hard and people used the video list in the index page at the bottom of the IDD PDF or used search function built in the browser. When they reached the GLIFOS page that actually presents the video, the participants found that the right side menu was very confusing. The preview part at the bottom was too small so that contents rarely fit in, and the contents were not automatically resized. To see the contents, users had to pop out the contents, but the pop-up button was not obvious and the pop-up often blocked the video. The right column menu also re-organizes whenever users click on an item. The clicked item is sent to the bottom as it shows the item in the preview window. However, it was very unintuitive so that the participants complained that an item disappeared when they clicked on it. Furthermore the layout functionality acted erroneously in Microsoft Internet Explorer.

13

Figure 4. GLIFOS video interface for the Midnight Address. Participants found that the menu on the right column looked useful, but it was very hard to use.

Overall, the participants found that the IDD is a very useful tool that has high quality multimedia contents, but it needed many fixes in the interface and the navigational tools.

14

Post-evaluation questionnaire

Figure 5. Post-evaluation questionnaire results. The participants thought the IDD was hard to find and use and the experience was not satisfactory.

Figure 5 shows the results of the post-evaluation questionnaire that asked if the experience using the IDD was satisfactory. The participants thought that it was very hard to find the IDD (2/5) from the LBJ library homepage. They were also negative about how easy it was to use the IDD (2.4/5). However, participants thought that acquiring target information on the IDD was not as hard (3.2/5). Participants were also very positive about the amount of the information on the IDD (4.6/5). Overall satisfaction was 2.5/5, and thus people thought the IDD needs some changes to improve the user experience. The reason for the dissatisfaction included that it was hard to navigate and find what they wanted, menus and icons were confusing, and the IDD was not well organized. Through the post-evaluation questionnaire, we could identify major complains for the current interface and collect a decent number of suggestions to improve the IDD on the LBJ library website as follows: Many participants thought that the homepage menus are good to navigate through the library website but not particularly to used the IDD. For the IDD page on the LBJ library website, participants thought that it had too much test, the link to the diary collection was confusing, and the IDD entry links were not clear. For the IDD PDF entries, the biggest complaints were about the KEY on the left column. It was not clear what KEY meant, and many people thought that was a navigational tool and tried to 15

click on it. On the other hand, the clickable icons on the diary entries were too small and many participants did not notice them. Participants also complained about the GLIFOS interface, and they thought the preview window was too small, pop-up interface was not intuitive, and the menu movements were very confusing. To improve the experience of using the IDD, participants suggested the following opinions: It is difficult to move between the diary entries. I would like a pop-out so you can return to the main page without using the back button on the browser. (P1) Menus are hidden, it would be better to have a site map or a search box. (P2) Easy links and better menus for multimedia (GLIFOS) (P3) Search capability and navigation improvements with in the IDD (P4, P5)

Discussion
Although each participant varied in their experiences and opinions of the Interactive Daily Diary tool, multiple common themes can be ascertained from feedback of the participants. We were able to compile these themes into three main problem areas that affect the usability of this tool. The first major problem encountered by users is the difficulty faced when attempting to find the Interactive Daily Diary from the LBJ Library main page. Participants did not know where to look when given the task of finding the IDD. Most participants browsed the dropdown menus to search for the IDD, because several of the menu headings (Education, Museum, Research Collections, and Online Exhibits) seemed likely places for the IDD to be located.

Figure 6. Menu options that might contain the IDD.

16

When the IDD was eventually found under Online Collections, users faced more confusing obstacles on the Interactive Diary main page. Several participants were confused by elements on this page, and accidentally navigated away from the IDD. The actual links to the Interactive Daily Diary are on the bottom of this page, and are easily missed. Two of the five participants had to have these links pointed out to them.

Figure 7. Distraction navigating to the IDD pages.

Whether or not users could find the IDD was a key stakeholder concern that this evaluation was meant to address. In a real situation, a user would not have a moderator instructing them to find the Interactive Diary; instead, they would have to come across it through browsing. This evaluation found that even with instruction, the users had difficulty getting to the IDD. If stakeholders want to highlight this tool in the future, designers will have to make it easier to find. The second major problem affecting the usability of this tool is the use of multiple, inconsistent interfaces. The user must interact with three different interfaces when attempting to access information from the IDD.

17

Figure 8. Multiple interfaces.

First, the user must select the diary entry he or she wants to view using a HTML page designed like the other pages on the LBJ Library site. Choosing an entry takes the user to a PDF interface. This PDF is either opened in the browser or downloaded, depending on the browser and browser settings. If the PDF is downloaded, this creates the problem of having to open a PDF reader in order to continue using the IDD. From the PDF interface, the user may link to other PDFs or the third interface, another webpage featuring information organized with GLIFOS. The use of different interfaces is not necessarily detrimental to the usability of a product. However, these interfaces have an inconsistent design between them. This makes them seem unrelated to each other and it is not obvious that they are all part of the same tool. Multiple interfaces also created navigation problems. Because a PDF is needed to display the scanned copy of the Diary, a program outside of the browser (such as Preview or Adobe Reader) must be used. This can derail the users natural navigation if she has to switch between applications. To make this tool more navigable, a similar interface utilizing predictable navigational options should be used throughout the parts of the tool. The final major usability obstacle is the use of confusing interactive elements within each interface. As alluded to earlier, the main page of the IDD has multiple navigational obstacles that mislead users. 18

The PDF Interface also has elements that confused multiple participants. The Key on the left side of the PDF pages looks a lot like a navigation menu. Four out of five participants (and both pilot participants) clicked on the Key, expecting it to link them somewhere. The real navigational links (the small icons embedded in the page) were over-looked at times.

Figure 9. Key looks like a navigation menu, small icons go unnoticed.

The GLIFOS page also contained many features with which users struggled. The most frequently noted issue with the GLIFOS page was the non-static file browser to the right of the video window. When participants click on one of the items, the menu acted in a manner unpredictable to them. The entire order of the items changes every time this menu is activated. When selected, an item appears in a small browser on the bottom of the menu, but many participants did not notice this. To properly view the intended item, a user must manipulate the browser window by popping it out and resizing it. This is awkward for the user, and doing so obstructs the view of other elements on the page.

19

Figure 10. File browser menu is not static.

GLIFOS offers the user the ability to manipulate the interface with which he or she is working. These features went mostly unnoticed by our participants. However, the ability to manipulate the interface is not essential to the usability of this product. What is essential is the ability to use the main features to find information. This evaluation showed that users struggled with the unfamiliar interface elements thus hindering their ability to properly use this tool.

Recommendations
Based on the analysis of observation data and users opinions, we suggest the following recommendations to improve the user experience of the IDD of the LBJ library website. Much of the usability problems of the Interactive Daily Diary stem from its navigational and design inconsistencies. These issues make the IDD tool less intuitive to users. This could lead to user frustration and eventual abandonment of the IDD. Incorporating the entire tool into one interface might solve many of the current usability problems of the IDD. By putting the IDD into a standalone interface, users will be less likely to confuse navigation of the IDD with navigation of the LBJ Library site. A standalone interface could also offer a consistent

20

and intuitive navigation between different sections of the IDD tool, which will eliminate jumping between applications. Figure 11 shows a mock-up web page for the IDD. We excluded confusing links on the left side and put the confusing link that links to the diary collection on the bottom of the page. After a short introduction, the entries are provided with pictures so that the users may intuitively understand that the entry links are the central contents of the IDD. The box and the horizontal scroll-bar implicitly suggests that the diary entries are organized in the chronological order, it allows user to navigate through multiple diary entries. It also allows the administrator to add entries without thinking about the space and re-arrangement.

Figure 11. Mock-up for the Interactive Daily Diary page of the LBJ library website. Confusing elements are deleted on the left side, and the diary entries take the central position with pictures so that the user would know intuitively that they are the primary contents of the page.

Figure 12 shows a mock-up page of an IDD entry. Note that it is now embedded to the website so that people use the same navigational frame instead of a PDF reader. On the left column, KEY was a very nebulous term and it was not clear whether the left column was an explanation of the icons or a navigation tool. We replaced it with Media and made it a navigation tool. When an icon is clicked, a 21

CSS layer pops up to show all the media in the entry of the day. In this way, we can make the left column into a navigation tool and get rid of the index page in the bottom.

Figure 12. An mock-up entry of the IDD. Instead of the ambiguous word KEY, we replaced it with Media and made the left column a clickable navigation tool.

Conclusion
The IDD is a powerful tool with useful information of hour-to-hour record of president Johnson. This tool provides a unique look into some of the most important days of a tumultuous era in American history. For history researchers and enthusiasts, this resource would provide very powerful and useful information that may help their research. However, the IDD has multiple usability issues that may need improvement to facilitate using it. Through this usability test, we have identified that there are some common complaints among our users about the IDD tool. The IDD is not organized intuitively, and people get easily confused and distracted by irrelevant links. Icons and navigational tools are confusing and do not follow the usual convention. We also collected suggestions to improve the IDD tool, and suggested recommendations and design improvements based on users opinions. Though we had only a 22

small number of participants (seven), we could find common complaints enough to identify a number of significant usability issues on the tool. In future, we wish we could scale up the test and find more meaningful suggestions to facilitate using this great tool.

23

Appendix A - Consent Form Participant name: Understanding Your Participation You have been asked to participate in a usability study about how websites meet your needs. Our purpose in conducting this study is to understand what makes some websites easier to use than others. In the session, well ask you to visit some websites and look for various things. Please keep in mind that this is a test of websites; we are not testing you. We may videotape all or some of the test for evaluation purposes but we will not release the recording to anybody. All information we collect concerning your participation in the session is confidential. Participation is voluntary. There are no physical or psychological risks associated with participating in this study. During the session, the study administrator will assist you and answer any questions. You may take short breaks as needed and may withdraw from this evaluation at any time. If you have any questions, you may ask now or at any time during the test. Statement of Informed Consent I have read the description of the study and I am aware of my rights as a participant. The conductor of the research study has assured me that my identity will remain anonymous and confidential. I agree to participate in the study.

Signature _______________________________

Date ___________________________________

24

Appendix B - Pre-evaluation Questionnaire 1. Have you ever visited LBJ Presidential Library website? [ Yes ] [ No ] 2. If you have visited LBJ website, what were you looking for in the website? [ ] Information about Lyndon B. Johnson [ ] Visitor Information [ ] Educational Information [ ] Research Information [ ] Online Exhibits (including the Interactive Daily Diary) [ ] Other Information (please specify) : __________________________________ 3. Have you ever used LBJ Interactive Daily Diary? [ Yes ] [ No ] 4. If so, could you find the information you were looking for with LBJ Interactive Daily Diary? [ Yes ] [ No ] 5. How would you rate yourself as an Internet user? [ Expert ] [ Good ] [ Beginner ] 6. Please check all that apply to you: [ [ [ [ ] Government official ] Program manager [ ] Researcher [ [ ] Student ] Faculty member ] IT professional

] Other ____________________________________

25

Appendix C - Exploratory Task This is an exploratory exercise. Please take a few minutes to navigate to the LBJ Interactive Daily Diary tool. Let the test administrator know if you have any trouble. After arriving at the Interactive Daily Diary tool, please take 2-3 minutes to just browse this web site. Please jot down your first impressions of this web site in the space provided below. Please be honest in your responses, your objective opinion will only support the purpose of this study. 1. What do you think the purpose of the Interactive Daily Diary tool is?

2. Who do you think is its target audience?

3. If this is the first visit of the Interactive Daily Diary, what would you say about this tool?

4. At a first glance, do you think this tool could provide you with any information of your interest?

26

Appendix D - Specific and open-ended tasks

Scenario: You are a secondary history teacher creating a series of lessons on topics associated with the Lyndon Johnson Administration. You need an image of LBJ for a presentation. Find any image of him as well as information about that image and its context. You are preparing a lesson on the beginning of the Vietnam War. Find the video of LBJs Midnight Address speech about the Gulf of Tonkin Incident. Explore the page and try to find some more information. Let us know what your thoughts are.

27

Appendix E - Post-evaluation questionnaire 1. What did you think about the menus you encountered?

2. What did you think of the icons on any of the pages?

3. How easy was it to navigate to the Interactive Daily Diary from the homepage? [Hard] 1 2 3 4 5 [Easy] 4. How easy was it to navigate the Interactive Daily Diary? [Hard] 1 2 3 4 5 [Easy] 5. What changes would you like to see done to the navigation?

6. How easy was it to get information from the Interactive Daily Diary? [Hard] 1 2 3 4 5 [Easy] 7. How would you rate the amount of text on the pages you saw? [Inappropriate] 1 2 3 4 5 [Good amount] 8. How satisfied are you with your experience with Interactive Daily Diary of LBJ Library? [Dissatisfied] 1 2 3 4 5 [Satisfied] 9. If you are not totally satisfied, please describe the reasons for your dissatisfaction below.

10. What other changes would you like to see done to the Interactive Daily Diary to make it easier to use?

28