Sie sind auf Seite 1von 17
MNSU Technical Communication Website Test Bernadette L. Cash November 28, 2016 1

MNSU Technical Communication Website Test

Bernadette L. Cash

November 28, 2016

1

Table of Contents Introduction 3 Executive Summary 3 Methodology 4 Sessions 4 Participants

Table of Contents

Introduction

3

Executive Summary

3

Methodology

4

Sessions

4

Participants

5

Results

6

Session 1: Card Sorting

6

Session 2: Interview

7

Overall Metrics

9

Recommendations

10

Conclusion

11

Appendices

12

Appendix A: Card Sorting Topics

12

Appendix B: List of Phone Interview Questions

13

Appendix C: Emails to Study Participants

15

2

Introduction The website at english.mnsu.edu/techcomm represents a subdiscipline of the English department at Minnesota

Introduction

The website at english.mnsu.edu/techcomm represents a subdiscipline of the English department at Minnesota State University, Mankato. This link provides information to current and prospective students about requirements, coursework, faculty, and resources for the school’s technical communication program.

A study was conducted to determine the usability issues that students have with the website. Testing was carried out remotely with a small group of university students. These users were asked to organize content into categories and then provide feedback about the website. Results from the sessions were noted and analyzed to identify suggested areas for improvement to the website.

The first remote testing session was administered using the OptimalSort feature from OptimalWorkshop.com. Sixteen topics from the current MNSU tech comm website served as online cards which users sorted into groups. The team leader recorded participants’ categories and their results.

The next session consisted of phone interviews between the test administrator and participants. By emailing instructions and hyperlinks beforehand, more information was obtained from participants regarding their card sorting results and about the tech comm website. The test administrator remotely interviewed participants while they viewed material on their own computer. Participants’ remarks, ratings, and responses were noted during the testing. (Throughout this document, the different roles of team leader, test administrator, and project leader all refer to the same person, Bernadette Cash.).

Executive Summary

This large usability study was conducted in two sessions. Card sorting occurred from November 9 through November 13, and interviews were held on November 16th and November 18. Based in central Illinois, the project leader conducted the testing online, over the phone or by using Skype to connect with participants located in Minnesota and Maryland. The purpose of the study was to assess the usability of the tech comm website’s information architecture and interface.

Four students participated in the testing. They were informed in advance that the time estimates for the card sorting and interviews would each be approximately 15 minutes long.

In general, most participants found that the MNSU tech comm website contains valuable information they needed, and that it is a source of reliable content. Participants who are tech comm majors consulted the website more frequently (as much as 8 times a month) than the minors (as little as 2–3 times total).

3

The usability study identified several issues including: • A need for organizing sidebar menu items

The usability study identified several issues including:

A need for organizing sidebar menu items into appropriate categories.

A need to check that all links function correctly.

A need to maintain consistency throughout the website.

A lack of modifications to accommodate students with disabilities.

A need to increase the font size and use high-contrast colors.

A possible consideration of updating the current vertical format to be concise and effective in its structure of content.

A suggestion to provide an overview of what students will learn

from tech comm coursework.

This document contains participant feedback, agreement ratings, task completion percentages, and recommendations for improvements. A copy of materials used in the testing session are included in the Attachments’ section.

Methodology

Sessions

Dr. Dawn Armfield, the ENG 466/566 professor, specifically arranged the team leader and participants for this usability study. Therefore, no pre-test questionnaires were given to recruit eligible participants.

The project leader sent emails to participants informing them of the study’s purpose and testing logistics, as well as requesting their timely participation. After they completed the card sorting, participants were invited to sign up for an interview during a time slot that was convenient for them.

Card sorting

Participants completed the card sorting in sessions that lasted anywhere from 8 to 19 minutes. The OptimalSort testing instructed participants to sort 16 topics (see Appendix A) into groups that made sense to them, and then they were to give each category a name.

Interview

Several days later participants were to call or use Skype for their interviews, which lasted an average of 20 minutes. The test administrator emailed each participant beforehand. The message contained specific information that participants would refer to during the testing. Participants were also asked to spend time looking at the tech comm home page and browse through the website.

The interview began with getting preliminary information, followed by a discussion (post-test) about participants’ card sorting results. Format preference In the next portion of the interview, participants were shown several examples of navigational menus from other schools’ tech comm websites. They offered their opinions on the different menu layouts and chose the format they preferred most for a potential tech comm website update. Evaluation Tasks/Scenarios These scenarios were created based on the project leader’s prior history of unsuccessful attempts in accessing information from specific parts of the tech

4

comm website. Consequently, the next part of the interview was to determine if participants achieved

comm website. Consequently, the next part of the interview was to determine if participants achieved similar results after completing the same tasks.

The test administrator first asked participants to identify how they knew if text on the home page was a hyperlink leading to additional information. Then they remarked on what they observed after doing the following tasks (see #7a–c in Appendix B):

Click on the Technical Communication text in the slideshow photos.

Click on the Spring 2014 Techniques newsletter link.

Click on the News & Events link.

Questionnaire The test administrator then asked participants to rank the interface on a

5-point Likert Scale ranging from Strongly Disagree to Strong Agree (see #9 in Appendix B). The following measures were based on Peter Morville’s User Experience Honeycomb:

a. This website has information that I need.

b. I think this website is easy to use.

c. I like how the design of this website looks.

d. I can find information quickly on this website.

e. People with disabilities would have no problems using this website.

f. This website is a dependable source of information.

Finally, the test administrator asked participants the following questions:

What did you like most about the website?

What did you like least about the website?

What recommendations do you have to improve the website?

See Appendix B for the full list of interview questions.

Participants

All participants were either senior or 4th-year undergraduate students in the Fall 2016 semester of ENG 466 Usability with Dr. Armfield at Minnesota State University, Mankato. The participants completed both testing sessions over a 9-day period from November 9th to November 18h. All four participants completed the card sorting and all four were interviewed by the test administrator. Of the four participants, three were male and one was female.

Involvement

As part of the preliminary information, participants described their connection to technical communication by mentioning their declared fields of study (see Table 1). How often participants used the website depended on their level of involvement with tech comm: those who are majors tend to visit the website more often than those who are minors.

Table 1: Under graduate involvement

tech comm

tech comm

tech comm

major

minor

certificate

2

2

0

5

Result s Session 1: Card Sorting All participants sorted the 16 topics into their own

Result s

Session 1: Card Sorting

All participants sorted the 16 topics into their own individual groups. However, nearly every participant preferred how the topics as they are currently presented on the tech comm website due to familiarity. Only one participant chose an individual preference over the arrangement currently displayed on the website. Also, no two participants’ results were exactly alike in how they were sorted, including the labels chosen to name the groups.

Three participants said that categorizing topics in accordance with their personal preference had limitations. First, they had difficulty with sorting leftover items that didn’t “fit” neatly into the groups they created. Secondly, three participants expressed that creating groups which made sense for their own individual situation meant a system that would not be applicable for all students.

Sorting patterns

Regardless of participants’ lack of confidence about individually creating an effective grouping system, there were some patterns from the results that may offer additional clues on how to categorize the topics.

The following items all had different group names and were sorted like this:

Bachelor’s degree Undergraduate minor Undergraduate certificate

Financial aid Admission requirements Bachelor’s degree Undergraduate certificate Undergraduate minor Courses

Master’s degree Publications Graduate certificate

Master’s degree Graduate certificate Capstone projects

Admission requirements Master’s degree Graduate certificate

Considering these specific lists, it makes sense to have at least two groups that differentiate appropriate topics into groups called Undergraduate and Graduate. In a similar manner, participants’ suggested labels of Help to Succeed and Helpful College Resources could simply be known as Resources.

Faculty was seen twice as an item in a category by itself with labels like Faculty/Staff Information and Faculty Information. Interestingly, most participants put Class Schedule and Courses in the same group, just as they currently are on the tech comm website. Participants agreed that Class Schedule was the one website item they used most frequently.

While there were no identical matches among the participants’ varied responses, there were a few potentially groups in common: Undergraduate, Graduate, Resources, Faculty, and Courses & Class Schedule.

6

Session 2: Interview Format preference Most participants picked a vertical navigational menu as their first

Session 2: Interview

Format preference

Most participants picked a vertical navigational menu as their first choice (see Table 2). Even though the MSU Mankato’s menu is also vertical, participants liked Michigan Tech’s menu better. They appreciated its clean and highly organized format: the use of main headings, subheadings, and subheading menu items. Also, compared to Michigan Tech, the items listed in the MSU Mankato menu are not general category headings, but, as one participant found, resemble a list of “too many headings on the side” (see Figure 1). It was suggested that some of these be grouped together and given category labels.

Figure 1: Menus, Michigan Tech vs. MSU Mankato

labels. Figure 1: Menus, Michigan Tech vs. MSU Mankato The group of tiles format was a
labels. Figure 1: Menus, Michigan Tech vs. MSU Mankato The group of tiles format was a

The group of tiles format was a hesitant second choice. Participants thought the photos with descriptive captions were helpful as visual category icons. However, they also critiqued it as having a “jumbled” appearance and looking “clustered and scattered.” Every participant gave the horizontal format the lowest rating for having “too few” categories that were also too broad in scope and looking very “blank,” causing a user to inefficiently search through all the drop-down menus.

Table 2: Participants’ format preferences

Website

1 st choice

2 nd choice

3 rd choice

Vertical

     

(Michigan Tech)

3

1

0

Horizontal (Utah State Univ)

0

0

4

Group of tiles (Univ of Minnesota)

1

3

0

Evaluation Tasks/Scenarios

All participants said they recognized hyperlinks on the home page because such text are typically underlined; one participant also added that a hyperlink is “highlighted” (changes a different color) when a user’s mouse hovers over

7

it. Using these identifying characteristics, the phrases indicated in all three tasks are technically supposed

it. Using these identifying characteristics, the phrases indicated in all three tasks are technically supposed to be hyperlinks.

A task was deemed successful if clicking on the specified link led to another page containing information (see column labeled “Successful hyperlink” in Table 3). Otherwise, the three other column labels describe what happened after a participant clicked on the text. There appears to be a high failure rate in working hyperlinks.

Only a single task was successful for one person. That participant could access information about the Spring 2014 Techniques newsletter; however, for everyone else, clicking on the hyperlink resulted in a session that eventually timed out.

Table 3: Summary of task responses

Task

Not a

Timed

Blank

Successful

Percent

link

out

page

hyperlink

success

A

4

0

0%

“Technical Communication”

B

3

1

25%

“Techniques newsletter”

C

4

0

0%

News & Events

Questionnaire

Using a 5-point Likert scale ranging from 1 to 5 (where 1 is Strongly Disagree, 3 is Neutral, and 5 is Strongly Agree), participants ranked MSU Mankato’s tech comm website interface. The statements used by participants to evaluate the website (see #9 in Appendix B) correspond to the factors from Morville’s User Experience Honeycomb.

U seful

Three-quarters of participants agreed that the tech comm website offered information that they needed. In fact, this represents the highest mean rating of all the statements in the questionnaire. Several participants mentioned that the “Current schedule” feature prompted them to use the website regularly. Credible Three-quarters of participants agreed they could trust the information presented on the tech comm website. Credibility ranked as the second highest mean rating in the questionnaire. Desirable Half the participants liked the tech comm website’s design. Two people remarked positively on the menu’s basic vertical format; one indifferent participant suggested that MSU Mankato keep the same basic vertical format but suggested an update like Michigan Tech’s layout. Usable, Findable Only one-quarter of participants found the tech comm website easy to use, making it the second lowest mean rating in the questionnaire. Regarding usability, one participant felt the nonfunctioning hyperlinks contributed to the website’s overall image of not being entirely usable. Similarly, only one-quarter of participants thought that information could be found quickly on the website. In fact, one critic resented having to spend “so much time clicking” to find things.

8

Accessible Unfortunately, the issues participants experienced with usability and findability resulted in no participants

Accessible Unfortunately, the issues participants experienced with usability and findability resulted in no participants agreeing that the website is particularly suitable for people with disabilities.

Overall Metrics

User experience

During the previous portion, participants rated six aspects of overall user experience (see Table 4), including measures of the website’s:

Usefulness

Usability

Desirability

Findability

Accessibility

Credibility

Seventy-five percent of participants agreed (i.e., agree or strongly agree) that the website has information they need; the same percentage also considered the information to be believable. With an average agreement rating of 2.5 or less, only 25% agreed that they could find information quickly and that the website was easy to use. No one (0%) felt that the website was accessible.

Table 4: Interview Question #9, parts a – f

     

Neither

       

Strongly

Disagree

Disagree

Agree or

Disagree

Agree

Strongly

Agree

Mean

Rating

Percent

Agree

The website has information I need.

   

1

2

1

4.0

75%

I

think this

             

website is easy to use.

1

1 1

1

2.5

25%

I

like how the

             

design of this website looks.

1 1

2

3.3

50%

I

can find

             

information

quickly on this website.

1

2

1

2.8

25%

People with disabilities would have no problems using this website.

1

1 2

     

2.3

0%

This website is a dependable source of information.

   

1

3

 

3.8

75%

Percent Agree (%) = Agree & Strongly Agree Responses combined

Likes, Dislikes, Participant Recommendations

After browsing the website on their own before the interview and completing the tasks and questionnaire, participants then provided feedback on what they liked most and least about the site, and recommendations for improving it. Liked Most The following items are what participants said they liked most:

slideshow of photos

9

• “the vertical aspect”; “color scheme matches the University’s”; home page at least tries to

“the vertical aspect”; “color scheme matches the University’s”; home page at least tries to “look professional”

information that students need (e.g., current schedule) is centrally

located and conveniently accessed from the home page Liked Least

The following comments capture what the participants liked the least:

finding things on the website is “like a maze”

One participant had to do a Google search for the school’s technical communication minor to find pertinent information.

information on website needs to be more “professional”: text should be direct in what it’s trying to say

purple hyperlinks need to “stand out more” from the gray background

(of the three columns on the home page)

text looks “too wordy”; lines of text look “cramped” Recommendations for Improvemen t

check all links for accuracy

use a bigger font for the lines of text

use white space more effectively

“consolidate text”; make text concise and “connected to tech comm majors”

have fewer headings in the left sidebar; headings need to be grouped

into categories

explain what students will learn from taking tech comm classes

Recommendations

A website should be accessible

Participants were asked to rank the website on accessibility.

Change

Justification

Severity

Enlarge the font size

None of the participants agreed that the website

High

was accessible. The website earned the lowest

Use a higher contrast for better readability

mean rating of 2.3 (out of 5) due to not being suitable for those with disabilities.

A website is easy to use

Participants were asked to rank the website on ease of use.

Change

Justification

Severity

Make headings and content more concise

Only 25% of participants agreed that the website was easy to use. For this aspect, the website

High

received its second lowest mean rating of 2.5

(out of 5).

Use white space more effectively

Participants said comments such as: text looked “cramped”; overall information is “too wordy.”

Information on a website can be found quickly Participants were asked to rank how quickly

Information on a website can be found quickly Participants were asked to rank how quickly information could be found on the website.

Change

Justification

Severity

Items in left sidebar should be grouped into categories

Just 25% of participants (half were neutral) agreed that it was quick to find information on

Moderate

Check links and content for accuracy

the website. In this regard, the website received

its the third lowest mean rating of 2.8 (out of

5).

Maintain consistency throughout the website

Nearly every participant was unsuccessful in using (what they expected to be) links that led to additional information.

A website is appealing to its users Participants were asked to rank how they liked the design of the website.

 

Change

Justification

Severity

Consider modifying the current vertical menu format to have more organization

Half of the participants liked the design elements, although 75% felt the current vertical format could be tweaked to be more appealing and

Moderate

arranged in a systematic way.

Provide an overview about the tech comm field

The participants who were tech comm minors thought it would be helpful to know what they would learn from tech comm classes.

Conclusion

Most participants felt that MSU Mankato’s tech comm website contains necessary information, and that its content is generally trustworthy. Even though the current website is serviceable for students’ needs, all participants agreed that there are areas of improvement. Revising the site to have improved organization, enhanced readability, accuracy, and a visually pleasing layout would do much to boost the overall user experience for a variety of students.

Appendices Appendix A: Card Sorting Topics Bachelor’s Degree Courses Internships Undergraduate Minor

Appendices

Appendix A: Card Sorting Topics

Bachelor’s Degree

Courses

Internships

Undergraduate Minor

Faculty

Organizations

Undergraduate

News & Events

Publications

Certificate

Admission

Class Schedule

Master’s Degree

Requirements

Techniques

Graduate Certificate

Capstone Projects

Financial Aid

Appendix B: List of Phone Interview Questions

Preliminary

1. Tell me what year of school you are in.

2. What is your major? Are you minoring in anything?

3. Why are you taking ENG 466 Usability?

Card Sorting

4. If you recall, the purpose of this usability study is to improve our school’s tech comm website. For the card sorting task, you sorted 16 topics taken from the website and arranged them into groups that made sense to you.

Here are your results:

[Screenshot of participant’s results inserted here.] Walk me through your reasoning about why you grouped items the way you did.

5. Before this interview, you were asked to look at the MSU Mankato tech comm website, http://english.mnsu.edu/techcomm/. Considering the way how topics are organized on the website vs. how you arranged the topics during the card sorting, which way do you prefer? Why?

Format preference

6. These are layout examples from other schools’ tech comm websites. If the tech comm website were to undergo a change, which navigation menu do you like the best:

vertical, horizontal, or a group of tiles?

Vertical (same type as current site) – http://www.mtu.edu/humanities/undergraduate/stc/

Horizontal – http://techcomm.usu.edu/

Group of tiles – http://cla.umn.edu/writing-studies/undergraduate

Tasks

7. Let’s go to the MSU Mankato tech comm website at the following link http://english.mnsu.edu/techcomm/. How can you tell if any specific text on this page is a hyperlink that leads to more information?

a. The words Technical Communication in the slideshow image are underlined. Tell me what happens when you click on that text.

b. Techniques is the name of the newsletter put out by the grad students. Tell me more about the Spring 2014 edition when you click on the link.

c. Tell me more about what you find out when you click on News & Events.

Questionnaire

8. Before this usability study, had you ever been to the tech comm website before? (If so, how often? If not, why not?)

9. For each of the following statements, please rate how you feel about it using a Likert scale of 1 to 5: Strongly Disagree, Disagree, Neither Agree or Disagree, Agree, or Strongly Agree.

a. This website has information that I need.

b. I think this website is easy to use.

c. I like how the design of this website looks.

d.

I can find information quickly on this website.

e. People with disabilities would have no problems using this website.

f. This website is a dependable source of information.

10. What about the website do you like the most?

11. What about the website do you like the least?

12. What suggestions do you have for improving the website?

Appendix C: Emails to Study Participants

Nov 8, 2016

Hi, everyone!

Group Recruiting email

This is Bernadette Cash, and Dr. Armfield has assigned you to be a participant in my usability study. I am looking for feedback that will help improve the technical communication website of Minnesota State University, Mankato. This is our school’s own tech comm page at http://english.mnsu.edu/techcomm/.

What will you be doing in the usability study? You will first be asked to do a card sorting task using topics from the tech comm website. In a later session, I will conduct an interview to learn more about your card sorting responses, as well as get your opinions, preferences, and suggestions about organizing content on the tech comm website.

How long will the testing take? About 15 minutes for card sorting, then 15 minutes for an interview.

When and where? Want to enjoy your Thanksgiving break? Me, too. Then let’s finish testing by Saturday, Nov 19! Here are my suggested time frames for getting things done:

§ Nov 8–13, Tue–Sun: Complete and submit the online card sorting task.

§ Nov 14–19, Mon–Sat: Meet with me for a phone interview.

I'll soon be emailing you a link for the first task, card sorting

. After I receive everyone's responses, I’ll

ask you to sign up for a phone interview with me next week. At that time, it would be helpful for you to have access to a computer when we talk.

Give your best effort As much as I would appreciate a quick turnaround, I also want this study to reflect accurate feedback. Take time to provide thoughtful responses. The results of this study will be valuable to share with the tech comm faculty.

Any questions? Feel free to email me at bernadette.cash@mnsu.edu.

Thanks so much for your help!

Bernadette Cash

Project Leader

Nov 8, 2016

Testing Session 1 emai l

Everyone, ready? Here is a link to the open card sorting session:

https://hdy78ys1.optimalworkshop.com/optimalsort/ajh8wyf7

Have fun!

Bernadette

Reminder and Intro for Testing Session 2 emai l

Nov 10, 2016

Just a friendly reminder to finish card sorting before 11:59pm THIS Sunday, Nov 13. Here is that link again: https://hdy78ys1.optimalworkshop.com/optimalsort/ajh8wyf7

Next step for the group: Check your schedule and select a time for a phone interview next week. I will give you access to the sign-up sheet soon.

Keep the following in mind:

§ Interviews are on the hour and half-hour but should take about 15 minutes.

§ Green = unavailable time blocks, so write your name in a white or gray blank.

§ I will wait on you to call me for your appt. The # to call is 309 XXX-XXXX, Central Time zone.

§ Have internet access on a computer when we talk.

§ I will ask questions about you and your responses to the card sorting task.

I will also ask for your feedback about our school's tech comm website, so

http://english.mnsu.edu/techcomm/

BEFORE the interview

.

be sure to look at
be sure to look
at

Thanks, and I look forward to talking with each of you soon.

Bernadette

Link for Testing Session 2 email

Nov 10, 2016

Everyone,

In case you didn't get the invitation to edit, here is the link to the interview sign-up sheet:

https://docs.google.com/spreadsheets/d/1HLBrAQ-

ZykkJC38El8f4odrMfb97QSknxBZjuZ7kkhY/edit?usp=sharing

Bernadette

Individual Testing Session 2 Reminder emai l ( sample )

Nov 15, 2016

Dear

,

This is a reminder and instructions for Wednesday, Nov 16 at 12:00pm Central Time.

§ Call 309 XXX-XXXX. If you would prefer, I can call you instead but you'll need to give me your number.

§ Please have internet access on a computer ready.

§ Take time to look through our school's tech comm website before we meet:

http://english.mnsu.edu/techcomm/

I will ask 12 questions. Below is an overview; I didn't list everything but this is what to expect. Have this email message open when we talk.

Looking forward to our interview! Thanks again for your help with this usability study.

Bernadette

Preliminary

#1

Your year in school

#2

Your major/minor

#3

Your reason for taking ENG 466

Card sorting

#4

Your results:

<Insert screenshot of participant’s card sorting results here.>

#5

MSU Mankato tech comm website – http://english.mnsu.edu/techcomm/

Format preference

#6

Which of the 3 do you like best? Layout examples from other schools' tech comm websites:

§ Vertical – http://www.mtu.edu/humanities/undergraduate/stc/

§ Horizontal – http://techcomm.usu.edu/

§ Group of tiles – http://cla.umn.edu/writing-studies/undergraduate

Tasks

#7

MSU Mankato tech comm website – http://english.mnsu.edu/techcomm/

a.

b.

c.

Questionnaire

#8

How often you've been to our school's tech comm website

#9

For the next 6 statements, use a Likert scale of 1 to 5:

1 – Strongly Disagree

2 – Disagree

3 – Neither Agree or Disagree

4 – Agree

5 – Strongly Agree

a)

b)

c)

d)

e)

f)

#10 What do you like most about the website? #11 What do you like the least about the website? #12 Suggestions for improvement?