Portable Beer Pong Scorer

Usability Test Report

Jon Woods and Group 6

Thomas Maxwell
Eric Mazzocco
Kyle Spahn
Brian Suessine
Introduction
The Portable Beer Pong Scorer is a small handheld device featuring a
touchscreen, wireless connectivity, and a novel interface by which an user with
an a priori acquaintance with the game of Beer Pong can score games for
purposes of record keeping. The game is traditionally scored with paper logs,
and must be transcribed to a digital form into a long-term database to keep track
of player statistics as is done with other major sports. This method is inefficient,
requiring far more steps than necessary to adequately store this recorded
information for easy retrieval, and limits the amount of data available. The
Portable Beer Pong Scorer is designed to mitigate the hindering concerns of
standard paper scoring by offering an intuitive and powerful interface to
increase the ease and effectiveness of recording this athletic competition.

The user test designed to record the effectiveness of the device was simply a
step-by-step comparison of an user's ability to track the progress of a game of
Beer Pong with a professional scorer's traditionally recorded logs. The
participant was given no instruction on how to use the device, nor any further
instructions beyond the description and purpose of the testing. This was to test
the intuitiveness of the device by requiring users to use their knowledge of the
scoring of the game of Beer Pong to control the device with as shallow a learning
curve as possible.

The Test
Participants were required to figure out that a 'rack' of cups must be set up in a
template before gameplay can begin. The user was to then figure out how to set
up the cups in the order by which they were arranged on the table. The
participant was then to allow progression of play, recording each player's hits,
misses, rearrangement of cups ('reracks'), and penalties until the point that one
team had won.

The purpose of this test was to determine if an user already familiar with the
traditional scoring of Beer Pong was able to pick up this device and immediately
use it for scoring purposes. There are several useful properties of this device
which, if proven to be intuitive and effective, would make it a powerful addition
to the sport of Beer Pong. One of the biggest advantages to a digital scorer is
that the information recorded is immediately available to send to the central
database without the need for transcription from paper. Additionally, there could
be additional levels of detail available for later retrieval, such as the actual cup
scored upon instead of simply the record of a score event.

The goals of test were to be evaluated by how successfully the participant could
figure out how to perform various tasks inherent in the scoring of a game of Beer
Pong. First, the participant was to be evaluated by how they organized the initial
placement of the cup layout compared to the layout template of the developers.
Then the participant was to be evaluated on how well they recorded the
statistical progression of the game; hits, misses, and correct cup identification
again compared to the professionally scored game. Penalties were to be
recorded, and evaluated by their correctness of player details of the event such
as the correct cup on which the penalty was perpetrated. Finally, in the event
that a team was left with three cups and requested a rerack, the participant was
to figure out how to record this event and was evaluated on whether they
successfully completed the event.

The Participants
The participants were divided into two groups for two sets of user tests. The first
test was held in a lab environment with three volunteer college students with
varying familiarity with the game of Beer Pong. The game was explained in detail
by the designers of the Scorer, as was the traditional method by which it was
scored. Two of the participants were paired off with two of the designers, which
served as two man teams for the gameplay progression. The third was asked to
score the game with no further instructions. The game progressed until
completion, and the scorer swapped positions with one of the participants who
had played in the last game. Participants were given water, and did not drink the
game fluids.

The secondary group of participants were semiprofessional Beer Pong players

active in a local league. This league was hosted by the Professional Scorer whose
results served as the comparison of results to the tested device. In these tests,
two man teams dueled in a sanctioned league game of Beer Pong, and an
individual from a third team acted as the scorer. The results recorded by the
Portable Beer Pong Scorer were not used as the final score of the games, and
will not be until it has been proven further in the field.

Testing Concerns
There were several testing considerations that had to be finalized before testing
could take place. The only functional component of the Portable Beer Pong
Scorer which is not guaranteed to be tested in a normal game is the penalty
feature. This only occurs when a cup is knocked from play by accident, and
results in the loss of a turn. This event typically takes place during later stages of
the game in which the dexterity of the players has been compromised. To
counteract this and force the testing of a participant's ability to correctly record
a penalty, one of the developers in gameplay would knock over one of their own
cups on purpose of the event did not occur on its own. This was not possible
during league play, and so this feature would only be tested on the control group
unless it occurred during one of the initial games of professionals.

Sobriety was also an issue. Under some circumstances, the individuals playing or
scoring a game of Beer Pong may be under the influence of age-appropriate
beverages. To ensure the most accurate results for the purposes of this test,
individuals in the control group were given water which they were not required
to consume. League play typically occurred with a best 3 of 5 scoring system,
which may have compromised the integrity of our data. Therefore, only the first
game of each round was tested with a sober participant acting as scorer.

Results
The results of the tests were fairly consistent, and showed several design
concerns as well as several considerations which were to be further evaluated
for future testing. In the initial design phase of cup layout, for example, every
single participant expressed difficulty with figuring out how to relate the layout
of the cups on the table into the device's cup layout designer. However, all
participants eventually were able to create a template of three rows, with one
cup on the first row, two on the second, and three on the third. This was
consistent with the actual layout of the cups, even if it was visually dissimilar.
One participant created two of these placements on the same template, even
though it was only designed to represent a single team's setup. He was stopped
and asked to start over and read the onscreen instructions.

The recording of game progression was consistent with the flow of actual game
play. The original intention of the device was that an player could have the
record of the specific up they hit during gameplay, which required a consistent
and standardized way of recording hits and misses. The biggest problem was
that each participant adopted their own system of representing the axes of play.
For example, if the leftmost cup were scored upon for a particular team, the
users varied in whether this correlated to the left or right most cup onscreen, on
whether to adopt a top-down or bottom-up approach to representation. This
meant that precise statistical recording was impossible, although participants
were consistent in their methodologies, and the game recording progressed in a
way that was 'visually correct' throughout play, if not 'statistically correct'.
Similarly, the cup chosen for a penalty was consistent with their chosen axis
representation, and the recording of the offending player was consistently
correct. No penalties occurred during league scoring, and they could not be
introduced due to it being a synthetic event in otherwise actual gameplay. When
a team was down to three cups and a rerack was requested, all participants were
able to identify the rerack button and its purposefulness, and the games
progressed from that point on.

Design Issues / Test Reconsiderations

The tests were designed to be as thorough as possible, forcing testing of all
possible functionality where applicable and surveying a wide sampling of users
with varying familiarity with the sport of Beer Pong. However, several issues
presented themselves during the testing procedure that illustrate several major
concerns with the design of the device and during our testing procedures.

The biggest issue that presented itself was in the recording of hits and misses. In
addition to the problems with unstandardized recording of left and right cups,
the users varied in their methodology of how to record a miss. The designers had
intended for the default action when clicking the next button without selecting a
scored upon cup as a miss. There were two concerns with this. First, in the event
of pressing the next button without selecting a scored cup, the participant was
greeted with virtually the same screen as if they had pressed the back button.
This resulted in the recording of misses and the recorded number of back clicks
to be almost useless, as there was no visually significant reason to choose one
over the other for similar tasks. Secondly, one user, not content to keep clicking
the next button to record misses, simply took no action to record misses.
Second, there was a lack of timing data available for this test which significantly
reduced the amount of quantitative analysis possible. The reasoning for this was
that the user's interaction with the device was dependent on outside factors,
namely the progression of a game of Beer Pong. Thus, there was no effective way
to discern how long it took an user to perform an event in relation to how long
the players took to progress from round to round. Additionally, there was original
intention to be able to record misses and errors, although the unintuitiveness of
the next and back buttons made this data unusable.

Third, the rerack posed a set of unique problems related to scoring. The device
keeps track of a running layout of cups per team. However, during a rerack
event, the device modifies the layout in memory to reflect the new layout. One
participant uncovered a bug that would not allow the history to progress back
beyond the rerack event.

Additionally, there were more pronounced differences in the two groups of

participants than was originally anticipated. In addition to not incurring any
penalties during league play, the games were also universally shorter as
experienced players make less errors than those just learning in a laboratory
environment. Furthermore, these semiprofessional Beer Pong players had a
much deeper familiarity with the scoring of the game than did the laboratory
participants, and therefore even made less mistakes when scoring the game.

The best representation of the experiences of the users may be the relative sizes
of the data accrued during testing. While not a specific measure of anything in
particular, it does represent the totality of the interactions that the participants
had with the entirety of the interface. The first lab participant generated 56K of
data, due in large part to unnecessarily clicking on buttons to determine their
functionality, and the large number of errors they made. The second lab
participant had similar interactions with 52K of data. Lab participant three had
the shortest data file of all at 16K, representative of their decision not to take
any action to represent misses and only record hits and penalties. The three field
participants all had similar sized interaction logs, at 28K, 24K, and 28K
respectively. This is largely representative of the short games they scored, and
their advanced familiarity of the traditional scoring systems and fewer errors.

Trouble Visually Visually Number

Trouble with with Back / Correct Correct ReRack Number of Back Data
Rack Next Game Penalty History of Button File
User Designer paradigm Progression Recorded Problems ReRacks Clicks Size
1 X X X X 2 4 56K
2 X X X X X 2 5 52K
3 X X X X 2 9 16K
4 X X 2 0 28K
5 X X X 1 1 24K
6 X X 1 0 28K
 User 1 User 2

User 3 User 4

User 5 User 6