Heuristic Evaluation

Whirled

Team Disco Mouse

 2

Table of Contents

Introduction……………………………………………………...………..…………………………………. 2
Methodology………………………………………………………………...………………………………. 2
Personas…………………………………………………………………………………………………….. 3
Scenarios…………………………………………………………………………………………………….. 4
Summary of findings .........….……………………………………………………………………….…….. 5
Details of Issues found..…………………………………………………………………………………..…6
Games: Issues 1 - 6…………………………….………………………………………………… 6
Help: Issues 7 & 8………………………………………………………………………………… 8
Chat: Issues 9 - 11…………………………………………...…………………………………… 8
Navigation: Issues 12 - 14………………………....…………………………………………... 12
Summary..........................................................................................................................................15
Appendix……………………………………………………………………………………………………..16
List of Heuristics….………...…………………………………………………………………… 16
Personas.............................................................................................................................17
Persona 1: Travis..................................................................................................17
Persona 2: Gloria..................................................................................................18
Persona 3: Wendy................................................................................................19
 3

Intro

The brainchild of Three Rings Design, a San Francisco based game developer, Whirled is a new
(alpha stage) online virtual world incorporating popular features including games, user-generated
content, a marketplace with an in-game currency, and social networking. Whirled will initially target
high school and college students based on the assumption that they will be more interested in
making things than a much younger or older audience. Eventually, Three Rings Design hopes to
attract a much broader demographic to Whirled.

This report details the findings we encountered during a heuristic evaluation of Whirled. A heuristic
evaluation, as suggested by Jakob Nielsen, involves multiple evaluators comparing the interface
against a list of recognized usability principles and best practices. Instances where the system
violates a usability principle are indications that there may be a usability problem.

Methodology

We began by identifying three scenarios and categories to base our evaluation around. We chose
to look at games, chatting and communicating, and adding avatars to the catalog. We decided on
the list of heuristics (which are detailed in the Appendix) and chose to prioritize the issues we
encountered using the following scale:

0 Not a problem

1 Minor usability issue; does not interfere with functionality

2 Major usability issue; causes some functionality problems

3 Usability catastrophe; severely hinders functionality

Once we had decided on the plan, each group members conducted an independent heuristic
evaluation of the scenarios. Each of us noted down issues we discovered, the heuristic number
that the issue violated, and assigned a preliminary priority on the 0 to 3 scale. This was done
separately so that we could cover the most ground, and also so that each team member could
identify their own separate issues without being influenced by others.

After the individual assessments, we reconvened, laid out the usability problems that we each had
found, and agreed on the issues that were most prominent, giving each issue a final priority ranking
from 0 to 3. At this point, we found that the upload avatar scenario was not as useful as the other
ones, so it was dropped it and other issues that we had encountered previously were added. Our
full collection of issues was broken into higher-level categories based on the nature of the problem,
rather than the task. We then assigned each section of the report to a different group member and
detailed what the specific issues were and why they were problems, using screenshots when
necessary.
 4
Personas Used

Gloria Kim
19 years old
College student (freshman, USC)
Lives in Los Angeles, CA
Extremely social (over 600 Facebook friends) – Gloria is almost
always in the company of several friends.
Hangs out at the Beverly Center, PinkBerry, new downtown hot
spots, and Little Tokyo
Drives a hand-me-down white Mercedes from Mom, but is angling for
a new, sportier ride.
Gloria sees online interactions as an extension, or enhancement, to
her real world interactions. Gloria is a communication hub. She will
occasionally play an online game, but she does it to be social.
Travis Cooper
17 years old
Junior at Badger High School
Lives in Milwaukee, Wisconsin with parents Donna and Roger
Has an older brother Logan, 22, and a younger sister Marie, 15
Enjoys playing Xbox 360, PC and NES games against school friends
Plays online games through Facebook
Travis loves the competition that online games provide him
His motivations are to beat his own scores and, more importantly, his
friends’ scores.

Wendy Zhu
24 years old
Graduate from SFSU, with a bachelors in fine art
Lives in Daly City, CA
Lives with her boyfriend Drew
Has a full-time job working at a Flax art store in San Francisco, she
also does freelance illustration and design on occasion
Used to play World of Warcraft with classmates at SFSU, but
stopped after the time commitment became too great
 5

Scenarios

The heuristic evaluation consisted of three scenarios performed by three different personas.
Travis, an avid game player, tried playing a game. Gloria, a fan of social networking, attempted to
chat. Wendy, an artist, uploaded and listed an avatar. We feel that these scenarios touched on all
of the desired user bases and personas for Whirled, without too much redundancy.

Scenario 1: Finding a game with Travis

In this scenario we tested how Travis, an avid game fan, would find a game and then invite
someone to play that game with him. Travis likes action games and is accustomed to console
game interfaces that are highly immersive. Because of this, we felt he would he would find
discontinuities in finding and joining a game problematic. We left how he found the game, and
whom he invited to play it with, open to help elucidate different aspects of the interface that could
use improving.

Scenario 2: Chatting with Gloria

With this scenario we wanted to look at how a Gloria might carry on a conversation in Whirled.
Gloria is very social and would probably want to carry on private and public conversations
simultaneously. Also, she is not very technologically adept, so any barriers to use would be
prohibitive to her.

Scenario 3: Uploading an avatar with Wendy

Wendy is an artist with a background in computer design who would like to show off her work.
Consequently, we choose her as the persona for the task of uploading an avatar. We felt that she,
as an artist, would be sensitive to the layout and design of the interface.
 6

Summary of Findings

Issue Severity Heuristic

Violated

1 No notification about players response to an invitation 2 1, 7

2 Difficult to add players to ongoing games 2 3, 5, 7

3 Selection of players for a game can only be done through the game 2 2, 3, 7

4 Game options are redundant 1 1, 8

5 Game’s landing page is cluttered 1 8

6 Having games in the catalog is confusing 1 9

7 Tutorials are inadequate 2 10

8 Chat help required using text commands 2 6

9 Unclear who a user can communicate with in a room 3 1, 2, 4, 6,

7, 8

10 Non-unified chat interface 2 1, 4, 8

11 Poor system visibility for communication functionality 2 1, 4

12 Exploring is difficult 3 2, 4

13 Re-findability is difficult 3 3, 4

14 Navigating is difficult 3 3, 4, 6, 9
 7

Games: Issues 1-6

Games are a fundamental portion of Whirled. They promote interaction among the users and
encourage frequent use. Because of the fundamental role they play in Whirled, their convenient
use should be of primary importance. While evaluating the site we found a number of issues with
games, in particular how groups are able to use the games. Issues 1 – 6 are all related to the use
of games and, in particular, multi-player game interactions and experiences

# Issue Severity Heuristics

Violated

1 No notification about players’ response to an invitation 2 1, 7

The games in Whirled have several issues that prevent them from being as enjoyable as they
could be. First, it is not possible to see if the other users are aware that you have invited them to
join a game. This means that you may wait for a user to join the game that isn’t even aware that
you are waiting.

# Issue Severity Heuristics

Violated

2 Difficult to add players to ongoing games 2 3, 5, 7

Once a game is started it is not possible to add other players. This means that if you either that
player gets left out or everyone needs to leave the game and then rejoin. In some instances this
may be a useful feature of the game, but this would be better to have an option allowing people to
add later.

# Issue Severity Heuristics

Violated

3 Selection of players for a game can only be done through 2 2, 3, 7

the game

The process of selecting both a game and a partner, or opponent, is poorly aligned with user’s
experience in the real world. In many situations it might be preferable, and more intuitive, to
reverse this order so that a user could, first, choose people to play with and, then, select a game.

# Issue Severity Heuristics

Violated

4 Game options are redundant 1 1, 8

Another feature of the games that may cause confusion stem from the buttons on the game’s
landing screen. Each game has four buttons that structure the options for that game. However,
most of the options are redundant. While this is not a major issue, it is unnecessary and would be
 8
better to leave the redundant buttons off.

# Issue Severity Heuristics

Violated

5 Game’s landing page is cluttered 1 8

Another problem with the game’s landing page it the amount of content presented there. Since the
users are probably clicking on the link to play the game, not find out more about it this page should
focus on the game play. All the information presented on this page makes it feel very cluttered
(Figure 1). To reduce this clutter it may be useful to have the game options on a different screen
from the game information.

Figure 1: Tree House defense landing page

# Issue Severity Heuristics

Violated

6 Having games in the catalog is confusing 1 9

Another issue with the games was confusion resulting from their presence in the catalog. By having
the games for sale and for free users may be confused about how to get to the games and as a
result may mistakenly buy games just to try them. The catalog needs to more explicitly state that
the games can be played without being purchased.
 9
Help: Issues 7 and 8

# Issue Severity Heuristics

Violated

7 Tutorials are inadequate 2 10

Several elements of the help features of Whirled seemed problematic as well. The amount of
documentation in the tutorials was insufficient to help users perform the tasks of interest.

# Issue Severity Heuristics

Violated

8 Chat help required using text commands 2 6

Also, using the help features for chatting requires knowing the correct key commands. While
having key commands for chatting is a convenient feature for some users, it is difficult for
beginners to remember and use correctly.

Chat: Issues 9-11

Our team identified a number of issues that specifically pertain to Whirled’s in-world chat and
communication system. As Whirled revolves heavily around interaction between users, both
publicly and privately, we feel it is important to make sure that communication makes sense. It is
especially important that communication be easy to pick up and learn, since new users will want to
start talking to others immediately without having to find help.

# Issue Severity Heuristics

Violated

9 Unclear who a user can communicate with in a room 3 1, 2, 4, 6, 7, 8

One of the most prominent issues we found with the chat system was that users may not realize
who else is in the room that they can talk to. This presents a tremendous block to communication,
since it inhibits a user’s ability to identify potential conversation partners and connections. We feel
that it represents one of the most important instances where the system status needs to be clearly
visible.

The first example of where this is unclear is the player list on the left (fig 2). Although this list does
communicate who is in the room effectively, there is still some confusion involved. The current
implementation of the player list shows a thumbnail of each user’s profile picture. However, a
user’s profile picture is not necessarily the same as the avatar that they are wearing. This creates
an inconsistency between what a user sees in the room and what they see in the player list. This
discrepancy causes the user additional cognitive load. Instead of being able to match pictures,
users must ignore the pictures and match names, which is less convenient. It forces them to
perform additional recall rather than recognition. Finally, users cannot interact with players through
this player list, which is less efficient for experienced users.
 10
A further inconsistency is that users who are currently playing games do not appear anywhere in
the current room, although they still show up in the user list. This will lead to confusion over
whether or not another user is available for chatting. Users may try to scour every corner of a room
looking for a user like this. Furthermore, since chatting is typically done through clicking on the
other user’s avatar, there is no way to contact a user in this situation at all.

Additionally, we feel that the use of pets presents a further problem because they interfere with a
user’s sense of presence about who is a room with them. Pets are items that users can buy in the
catalog, and then place in a room. Once placed, they will take the form of an avatar and then
automatically start to move and interact with other items in the room. For instance, there are
monsters that express emotions and even use animations that suggest that they are attacking
users. Given that they move by themselves and seem to understand where others are, it is easy to
confuse pets with other users. Whirled has provided some subtle cues indicating which avatars
are pets and which are users, such as by using white text for a pet and blue text for a user, but we
feel this is not enough to convey the difference to a user.

In particular, we feel that pets present an extremely confusing situation for a new user. Pets are
not mentioned during the initial tour, or in any of the help text or tutorials. Their existence can
basically only be discovered by looking in the Catalog and seeing that pets are a section of the
catalog. Even then, new users may not know what pets look like and how they act. As a result,
new users may assume that any object they run into that seems to be interactive is another person.

Figure 2: In-World User List. Note the “Finger puppet” picture used for both players, despite their
different avatars, and the pet "Bunny"

# Issue Severity Heuristics

Violated

10 Non-unified chat interface 2 1, 4, 8

We feel that the chat screen interface is not well-unified. The chat interface consists of the user list
indicated above, a message bar and send button on the bottom toolbar, and a log of messages on
the left side. Although each of these elements is located on the left side of the screen, we feel that
 11
they are not held together strongly enough to suggest that they are all part of the same system.
The message log is also intermixed with system messages, such as notifications of users signing
off, help text, and confirmations of user actions. The result is that the user is not aware of the
current status of the conversations that he is having.

Figure 3: The chat interface, with player list, chat text and message bar in red

# Issue Severity Heuristics

Violated

11 Poor system visibility for communication functionality 2 1, 4

We feel that there are inadequate notifications for the chat system. There are no audio cues or
sounds played when a new message is received, either in the public chat system or the private
chat system. This means that if a user is not actively looking at their screen, they could miss
graphic cues like text bubbles.

The private chat system is the most problematic. The only indication that someone is trying to start
a private chat with you is that a new tab appears on the top of the screen, as indicated in Figure 4.
This tab also pulsates when a message is sent in a private conversation. In addition to the fact that
the tab is small and easy to overlook, it is also in a screen area that does not get much visibility.
 12

Figure 4: Location of new chat tab

Furthermore, the tab metaphor is poorly implemented. Tabs suggest that the entire screen should
change when a new tab is clicked, but this does not happen with Whirled’s tabs. In this case, the
only thing that changes with tabs is that the public messages disappear and the private messages
reappear. It’s also not clear that the same message bar is used to send messages to both public
conversations and private chats.

Finally, the use of colors in the tabs is inconsistent. Rooms use black or dark grey tabs, whereas
private chats use blue tabs. This makes it difficult to tell which tab is currently selected, especially
in cases when a user has two tabs open. The diagrams below show how unclear the situation is.

Figure 5: Private chat tab selected

Figure 6: Room tab selected

 13
Navigation: Issues 12-14

# Issue Severity Heuristics

Violated

12 Exploring is difficult 3 2, 4

Whirled is difficult to explore. There is no map of the virtual world that could allow users to develop
a mental model of the environment, and thereby establish a sense of scale and place. The default
linking navigation from a user's "home" or room is one door connecting to the room of the "friend"
that invited the user. The friend's room then has a door to a common room, which in turn affords
access to other areas of Whirled. This creates an odd railroad-apartment effect requiring a user to
pass through a friend’s room to go anywhere from their own room.

From the common room, one navigation option is an "explore" link. This leads to a seemingly
endless series of rooms connected by doors, but is otherwise lacking in any continuity of place. As
a user explores, there are no indicators of her path to help her orient herself, such as the
“breadcrumb” trails that are often found on web sites.

# Issue Severity Heuristics

Violated

13 Re-findability is difficult 3 3, 4

Whirled doesn’t have any path or visit history available to users or a "save" feature that would allow
a user to return to a place of interest within Whirled. Rooms can be bookmarked with a browser’s
bookmark function, just like a web page, but users are not aware of this. Because of the rich
media environment, users may come to assume that basic browser functions will be disabled.
Furthermore, accessing browser functions breaks the in-game, virtual world metaphor. People are
accustomed to virtual worlds having their complete functionality contained within the game
environment.

# Issue Severity Heuristics

Violated

14 Navigating is difficult 3 3, 4, 6, 9
 14
Whirled presents users with many navigation challenges. The site needs a comprehensive review
of its fundamental information architecture. Three example tasks will be used to illustrate the
navigation issues: (1) navigating back; (2) navigating to play a game; and (3) navigating to
information about creating content for Whirled.

Navigating back

When arriving in a room in Whirled, there may or may not be a "return" door allowing for navigation
back to the previous room. Back navigation is a critical function for users central to Nielsen’s
heuristic #3, which says that users should have the control to recover from errors, or undo their
actions. There are two global options for back navigation within Whirled, but users may not be
aware of either of them.
One option for back navigation is the browser’s back button, but this has the same drawback
previously discussed with respect to using the bookmark feature – it is non-intuitive because of
other rich-media experiences and it breaks with conventions of other virtual world experiences. A
second option for back navigation (Figure 7) is the very small back button that the Whirled
developers have included to the far right of Whirled’s bottom frame, which is basically a tool bar.
Users may not discover this button, particularly as it is not located where users are accustomed to
looking for back navigation, which is in the upper left corner on web browsers.

Fig. 7: Back button in lower right hand corner of a browser window

 15
Navigating to play a game

Navigating a path within the metaphor of the virtual world to initiate game play is difficult and
confusing. A user must navigate to the common room, Brave New World, and then link on the
“games” sign or the “games” doorway. An alternative is to navigate to games by using the global
navigation buttons and menus, which is the option suggested by Whirled’s “help”. The problem is
that the navigation path to games is to select “Places/Whirledwide” and the user must know this,
just as they needed to know to navigate to the common room in the first example.

Fig. 8: Whirledwide with “top games” on the far left and the “All Games” link at the bottom, lower left

From Whirledwide, there are further problems with the navigation to games. First, the page design
lists “top games”, not all the games, and is constrained by the space available on the left hand side
(figure 7). Furthermore, the “All Games” link on the bottom links to the catalog of games for
purchase, which is not the correct page/interface for initiating game play. Clicking on a game in the
catalog has a completely different result from clicking on the same game if it was presented in the
top game list on Whirledwide.

Navigating to information about creating content

It is difficult to find Whirled’s user-created content areas, which contain information about what a
user can create (avatars, furniture, etc.) as well as the actual interfaces for uploading and tweaking
creations. These areas are located under the "My Stuff" menu, and then under subheadings
according to content type (avatars, furniture, etc.). To make an avatar, for example, a user should
navigate to "my stuff"/"avatars". The “my stuff” language is confusing and unintuitive when
combined with the idea of creation, or learning about creation. “My stuff” suggests a place for
things that a user already has.
 16
Summary

By conducting our heuristic evaluation, we were able to identify fourteen potential usability issues,
including issues with playing games, getting help, communicating and navigating. While the
majority of these issues were rather minor, we did find some that were significant and pervasive. Of
the areas examined, we found the most issues with navigation and communication. In particular,
the lack of clarity of which other users are available for chatting and the difficulty in finding and re-
finding interesting places in the virtual world seem the most problematic. Because the components
of navigating and communicating are so fundamental to the action of users in Whirled, we feel that
addressing them is imperative. With these issues resolved, we feel that user enjoyment of Whirled
would dramatically improve.
 17

Appendix

List of Usability Heuristics

We used a list of heuristics proposed for rich Internet applications, as suggested by Boxes and
Arrows(http://www.boxesandarrows.com/view/usability_heuristics_for_rich_internet_applications).
They are largely based off a list provided by Jakob Nielsen.

1. Visibility of System Status

The system should give some indication of what it is doing and what the current state of the
application is. This includes expressing when the system is busy, and whether user input
has been received successfully.

2. Match between System and Real World

The system should use metaphors and language that make sense to the user, rather than
making use of implementation details.

3. User Control and Freedom

The system should allow users to recover from mistakes by means of undo and redo.

4. Consistency and Standards

The system should have consistent meanings for terms and use documented design
standards when possible.

5. Error Prevention
The system should be designed so that it is impossible for a user to make an error.

6. Recognition rather than Recall

The system should provide the user with relevant options instead of forcing him to
remember and recall important information.

7. Flexibility and Efficiency of Use

The system should support multiple ways of performing a task so that experienced users
can use more efficient methods.

8. Aesthetic and Minimalist Design

The system should only display information that is relevant to the task at hand.

9. Help users recognize, diagnose, and recover from errors

The system should help the user understand when an error has occurred by
presenting the error messages in plain language, indicate the problem and constructively
suggest a solution.

10. Help and Documentation

While systems should be designed so that no additional resources are necessary to operate
them, when help is needed it should be, easy to search, focused on the user’s task, consists
of a list concrete steps to carry out and not be too large.
 18
Personas used

Persona 1
Travis Cooper
Male, 17 years old
Junior at Badger High School
Lives in Milwaukee, Wisconsin with parents Donna and Roger
Has an older brother Logan, 22, and a younger sister Marie, 15

Travis likes the challenge and competition that video games bring him. He loves nothing more
than taking on his friends from school and laying the smack down. He is infamous for playfully
needling the loser of last night’s match, or grudgingly admitting his failures if he happened to be on
the losing side. Keeping track of his scores compared to his friends’ scores is a major motivating
factor for him. If no one else is around, Travis can be found playing a game by himself if he feels it
has something for him to achieve, like a high score, a better weapon or a faster way through the
game. However, even though he will devote a lot of time to a game he likes, he will tend not to
bother if he feels it’s too hard to get into. He can be heard saying, “It’s not worth the effort!” if he
gets frustrated with learning a game.

Travis grew up with video games and is familiar with everything from online PC games to new Xbox
360 titles. He has an interest in computers and social networking sites, but mainly as an adjunct to
his interest in games. Logan invited him to Facebook, and he took to it immediately once he saw
how many games he could play with his friends. Additionally, he once tried to learn C++
programming so he could make his own RPG, but gave up once he realized how much work it
would be. He knows some basic HTML and used it to construct a simple webpage for his guild.

Travis has a fairly typical high school life. He plays the trombone in the concert band, and is on the
soccer and track teams. He likes to go snowboarding in the winter, and the first snow of the year is
almost an annual event for him and his friends. He takes road trips to visit Logan at Madison, and
sometimes goes on ski trips with his family.
 19

Persona 2
Gloria
Korean-American
Female
College student (freshman, USC)
19 years old
Los Angeles, CA

Gloria is extremely social. She is a college student at USC, living in new freshman housing. Gloria
is almost always with at least one friend, and more often than not with several friends. They study
together, work out together, hang out together, and shop together. Dating is a group activity. Cell
phones are appendages with texting and photo-sharing a constant background activity. Gloria has
over 600 friends on Facebook and is also active on MySpace (but, less so).

Hangouts – the Beverly Center, 3rd Street, Koreatown (K-town) for Korean food and Soju,
PinkBerry (yogurt), increasingly downtown LA since new places are opening all the time, and Little
Tokyo has always been a favorite. Car – hand-me-down Mercedes sedan (white) from Mom, but
her parents have agreed to a new car this year if Gloria’s grades are high enough.

Gloria’s family lives on the west side of LA. Her transition to USC was very easy – she still sees
her high school friends, and she has great new college friends, including several who she knew
prior to starting USC. She can go home whenever she wants. Gloria is close to her parents,
especially her mother who was very happy when Gloria chose a college in Los Angeles. Mother
and daughter often meet up for shopping and a meal when Gloria has a break from her studies.
Gloria has an older brother James who is pre-med at UC Berkeley. She and James get along well,
and she enjoys his visits. Gloria and her friends will join James and his friends out on the town.

Just like her cell phone, Gloria sees online interactions/communications as an extension, or
enhancement, to her real world interactions. Gloria is a communication hub whether in person,
online, by cell phone, etc. She will occasionally play an online game, but her motivations in doing
so are social, not boredom. If she has time to spare, she’d rather check in with friends than play a
game.
 20

Persona 3
Wendy Zhu
24 years old
Graduate from SFSU, with a bachelors in fine art
Lives in Daly City, CA
Lives with her boyfriend Drew
Has a fulltime job working at a Flax art store in San Francisco, she also does freelance illustration
and design on occasion.

Wendy graduated last year from San Francisco State college with a degree in fine arts, specializing
in Conceptual/Information Arts. She owes $15,000 in student loans, the rent on her studio
apartment is $1,100 (which she splits with Drew) and she makes $9.50 an hour at Flax. She also
commutes to work spending $8.00 roundtrip on BART and MUNI. Drew works as a web designer
and occasionally helps Wendy get illustration jobs for his clients.

Wendy has a Black MacBook that she got as a graduation gift from her family. This is her second
Mac laptop; her previous laptop was an iBook that she got as a freshman in college. She connects
to the Internet using the wireless signal of her neighbor’s apartment. Consequently her Internet
connection is slow and intermittent. When online, she uses Firefox to check her FaceBook account
and her Gmail account. She has over 500 friends on FaceBook, most of whom are classmates she
has never spoken to.

When she was in college she played World of Warcraft regularly. Her character was a Jeweler and
she enjoyed making things in the game more than the combat. Wendy reached a high level before
she got tired of the amount of time the game consumed. When playing WoW she would usually
meet with her friends from school at one of their dorms and play on weekend evenings. She
enjoyed the social aspect of playing the game as well as customizing her character. Wendy would
still enjoy playing games online with her friends, but doesn’t want to invest the time or money
necessary to play a MMORPG.

Wendy is interested in trying to develop avatars for Second Life or a similar world. She thinks this
might be a good way to combine her art background and her love of gaming. She knows that it is
possible to make a good living designing and selling avatars in Second Life and would like to learn
how to get started.