Amtrak.

com Usability Test Kit 

Group 3: ​Ruby Davis, Sierramatice Karras, Reine Abubakar, Connor Garrett 
November 9th, 2018 

Description 
The website aims to promote their business by providing information about their services, as well 
as facilitating the ticket-booking and trip-planning process. However, key features of of the site 
are lacking in their ease of use and efficiency. The goals of this usability study are to assess the 
process of purchasing tickets and researching relevant trip information and find ways to improve 
these common task flows. Usability tests can identify areas of confusion, speed bumps, and 
common errors. Improving the usability of these features will make for a more pleasant and 
efficient experience, encouraging users to purchase tickets and return for future trips, thus 
increasing Amtrak’s revenue and consumer loyalty.  

Research Questions 2 

Researcher Guide 2 
Sections to Print Before Testing 2 

Observer Guide 3 

Screening Questionnaire 3 
Qualifications for Study Entry 5 

Consent Form 6 

Facilitation Scripts 7 

Pre-Test Questionnaire 8 

Post-session Questionnaire 8 

Post-Task Questionnaires 9 

Moderator/Observer Task Scenarios 9 

Task 1: One way ticket 9 
Task 2: Multi Trip Tickets 10 
Task 3: Search Feature 10 
Task 4: Train Tracking 10 

Participant Task Scenarios 12 

Data Logging and Note Taking 16 

Research Questions 
1. Are the users able to successfully purchase tickets for their specific needs?  
2. Can users find the the most efficient ticket for their itinerary at the best price? 
3. Can users get through common tasks without reaching out for assistance?  
4. How should the site be navigationally organized to improve efficiency for inexperienced 
Amtrak customers?  
5. Can users fully comprehend the train tracking information without difficulty? How quickly, 
easily, and successfully can users navigate between the different tracking systems to find 
the information they need?  
6. Are users able to successfully use the search feature? 

Researcher Guide 
Participant Confirmation 
● Send confirmation message or calendar invite once participant is scheduled with: 
○ Date & time 
○ Study purpose 
○ Study summary 
○ Location of session 
● Contact participants one day before scheduled test with reminder message 
● Contact participant one hour before scheduled test with reminder message 

Sections to Print Before Testing 

- (1) Scripts 
- (1) Task List Moderator Version 
- (1) Verbal Questionnaire Script 
- (9) Consent Form 
- (9) Task List Participant Version 
 
Computer Set-Up Before Testing  
Bring two laptops to the session. Ensure that the following tasks are complete: 
● Participant laptop is cleared of all other windows or notifications 
● Both laptops plugged in (this ensures they won’t run out of battery during the test) 
● Participant laptop webcam is not obscured 
● Participant laptop microphone successfully registers audio 
● UserZoom session is ready for activation on researcher laptop 
● UserZoom is installed on participant laptop 
● UserZoom session link is loaded on participant laptop 
 ● Amtrak website is open on web browser on participant laptop 
● Set up participant laptop so that both participant and moderator can see the screen, 
additionally the observer should be able to see if possible.  

Observer Guide 
Supplies  
● Notebook  
● Pen 
● Data Logging and Note Taking Sheet 
● Timer 
 
Notes  
During session, record the following in the Airtable spreadsheet. 
● Time on each task 
● Number of errors during task 
○ Errors are defined as when a user clicks on the wrong thing or needs to backtrack 
to successfully complete a task.  
● Whether the participant was able to successfully complete each task 
● Number of assists per task 
● Relevant or interesting quotes from participants 
● Perceived participant emotional responses (frustration, confusion, excitement) 
● Physical gestures or body language of participants during different tasks 
● Either moderator can interrupt, given that it doesn’t complicate the completion of a task. 

Screening Questionnaire 
Thank you for your interest in our research study! Please answer the following questions as 
honestly as possible. The study will be between 30 and 60 minutes long and will take place on 
the University of Washington Seattle Campus between November 11th and 17th 
 
Your answers personal information will not be shared outside of the researchers conducting this 
study. Any answers you provide to our questions will be anonymized before it is presented in our 
report. 
 
1] How old are you?  
______ 
 
2] What gender do you identify as? 
 ______ 
 
3] Given your best estimate, how many times did you travel in the last year?  
______ 
 
4] Do you typically use a travel agent to plan your trips?  
- Yes (exit survey) 
- No 
 
5] Have you ever or do you currently work in the travel industry? 
- Yes (exit survey) 
- No 
 
6] Have you ever or do you currently work in UX/UI?  
- Yes (exit survey) 
- No 
 
7] How frequently do you travel by train?  
- Often 
- Sometimes 
- Rarely 
- Never (skip question 7a) 
 
7a] What was the purpose of your train rides?  
- Commuting 
- Domestic Vacation  
- International Vacation 
- Other 
 
8] Which of the following train ticket sites have you used in the past? (Select all that apply)  
[randomize order in survey] 
- Soundtransit.org (skip to question 9) 
- Wanderu.com (skip to question 9) 
- Amtrak.com (continue to questions 8a - 8d) 
- Raileurope.com (skip to question 9) 
- Other 
 
8a] Approximately how many times have you visited Amtrak.com in the last year?  
_____ 
 
8b] Did you use the site to book a ticket?  
- Yes 
- No 
 
 
9] Would you be available to meet for a one hour study at the University of Washington Seattle 
Campus on any of the following days? (select all that might work for you) 
- Sunday November 11th 
- Monday November 12th 
- Tuesday November 13th 
- Thursday November 15th 
- None (exit) 
 
10] Please leave your name, phone, and email address. 
Name _________________ 
Phone ________________ 
Email _________________ 
 
Thank you for filling out our survey! We really appreciate you taking your time to help us with our 
research. 
 
We’ll take a look at your answers to this survey, and if it looks like you’ll be a good fit for our 
study, we’ll be in touch soon to schedule a session.  
 
Thanks again, and have a great day! 

Qualifications for Study Entry

 
Question 1: Participants should be between 18 and 70. 
 
Question 2: We would like to recruit in a proportion of 50% men, 50% women but this can be 
adjusted based on available participants.  
 
Question 3: Participants should have traveled at least twice in the last year. 
 
Questions 4-6: Participants should not use a travel agent, work in the travel agency or in UX/UI. 
 
Question 9: Participants should have used the Amtrak site less than 5 times in the past year. 
 
Question 10-11: Participants need to be able to come to campus for the study and provide contact 
information. 
Consent Form 
Participant: ________ 
 
I agree to participate in the study conducted by the Amtrak.com Usability Test Group at the 
University of Washington, Human Centered Design and Engineering Department. 
 
During this study: 
- I will be asked to perform certain tasks on a computer 
- I will be interviewed regarding the tasks I’ve performed 
- I will be recorded through audio and video during the session 
 
I understand and consent to the use and release of the recording by the Amtrak.com Usability 
Test Group at the University of Washington, Human Centered Design and Engineering 
Department. I understand that the information and recording is for research purposes only and 
that my name and image will not be used for any other purpose. I relinquish any rights to the 
recording used by Amtrak.com Usability Test Group at the University of Washington, Human 
Centered Design and Engineering Department without further permission. 
 
Participation in this usability study is voluntary. All information and recordings will remain strictly 
confidential. The descriptions and findings may be used to help improve the Amtrak.com website 
design. At no time will my name or any other identification be used. I have the ability to withdraw 
consent to the experiment and stop participation at any time. 
 
The recordings are for research purposes only and will only be used by the involved students in 
class and portfolio presentations.  
 
Below is my signature indicating that I have read and understood the information on this form and 
that any questions I might have about the session have been answered. 
 
Date:_____________ 
 
 
Participant’s printed name: __________________________________________________ 
 
 
Participant’s signature : ____________________________________________________ 
 
Thank you, we appreciate your participation and your time. 

   
Facilitation Scripts 
Introductory Task Before Participant Enters 
❏ Have UserZoom open and ready to go on participant and researcher laptops 
❏ Start and then pause recording before welcoming participant into the room 
 
“Hello! Thank you for coming. Today we will be conducting a usability study of the Amtrak 
website. I will be walking you through the session today. My research team member is here to 
make sure we are accurately recording the session and they will be taking notes along the way. I 
will be reading from a script to ensure my instructions to every participant are the same.”  
 
“We've asked you to come here to try using the Amtrak.com website so we can see whether or 
not it works as intended. I want to make it clear that we're testing the site, not you, the user. 
There are no wrong or right answers.” 
 
“None of us had any part on the design process for this site and our objective is to improve it so 
we need to know what you honestly think. You won’t hurt any of our feelings, and it would be 
very helpful for us if you just speak your thoughts and gave your full opinion.” 
 
“As we go along, I'm going to ask you to think out loud, to tell me what's going through your mind 
as you complete tasks. If you have any questions as we go along, please ask them. I may not be 
able to answer them right away, but I will take note of it and see if I can answer it at the end of the 
session. If you still have any questions when we’re done I’ll try to answer them then.”  
 
“If you need to take a break at any point, just let me know. With your permission, we're going to 
record your computer screen and audio.”  
 
“We have a consent form for you to sign today. This will allow us to record the session for later 
review. The video will be used only to help us figure out how to improve the site, and it will only 
be seen by people working on the project and in result presentations.” 
 
“Now I will explain how the session will work, but do you have any questions so far?”  
 
“We will start the session by reading a few scenarios and asking you to complete some tasks. 
After each task, we will ask you a couple of questions about your experience performing the task. 
Are you ready to begin?” 
 
Post-Script Tasks 
❏ Have the participant sign the consent form 
❏ Start the UserZoom Study (audio, video recording) 
❏ Thinking aloud warm up (practice speaking as they click around) 

Pre-Test Questionnaire 
[Verbally ask the participant, Note-taker will record notes] 
 
1. How frequently do you travel?   
2. How often do you travel by rail?  
3. Could you tell us about your experience with booking transportation for a previous trip 
(including planes, cars, etc.)? 
4. Do you typically travel with luggage? 
5. When was the last time you used Amtrak.com? What was the experience like?  
6. Do you believe that travelling by rail is a difficult experience? 
7. Why do you choose your preferred mode of travel? 

Post-session Questionnaire
[Verbally ask the participant, Note-taker will record notes] 
 
1. What do you like about the website? Why? 
2. What do you dislike about the website? Why? 
a. Is there anything you found particularly frustrating? 
3. Would you be confident booking a ticket alone? Are there any features you felt were 
missing from the website? 
4. If you could change one thing about this site what would it be? 
5. Is there anything that we could do to improve the participant experience for our study? 
Post-Task Questionnaires 
For each task (distributed on the scenario sheets) 
1. How easy or difficult was it to complete the task?  
Easy Difficult 
1 ------------- 2 --------------- 3 --------------- 4 ---------------- 5 
2. How understandable or confusing was the process of completing this task? 
Understandable Confusing  
1 ------------- 2 --------------- 3 --------------- 4 ---------------- 5 
3.  Anything else you want to tell us? 
 
Verbally probe for more specifics on feelings of task (“Why did you give this rating?”) 

Moderator/Observer Task Scenarios 

Moderator will read scenarios outloud to the participant. The participant will also have a copy to 
reference that will include post task questions.  

Task 1: One way ticket 

Goal​​: Starting on the Home Page, search for train tickets that fit a given scenario covering 
different types of travelers and desires for the route (bike carry-on, price, length of trip). Compare 
the tickets to find either the the fastest, best deal, or least transfers as specified in the senario. 
Starting Condition:​​ Home Page 
Success​​: Find a ticket that meets the given criteria 
Fail​​: The user is unable to purchase a ticket with for the route with the specific constraints 
Needs​​: Train routes that we know have the necessary amenities for a given situation 
(dogs and bikes are not allowed on all routes)   
Scenario​​:   
You and a friend want to travel to Portland, Oregon on Friday, November 30th for a concert the 
following day. You will be leaving from Seattle, Washington. You would like to find the best ticket. 
(Half of users will be given A and half will be given B) 
Scenario A: ​Your friend uses a wheelchair that may be stowed during travel. This friend 
also has a small service dog that will assist them. 
 Scenario B: ​Your friend is a member of the military. Both you and your friend will be 
transporting your bikes with you on your trip. 

Task 2: Multi Trip Tickets 

Goal​​: Find a multi-city route that fits a given scenario covering different types of travelers and 
desires for the route. 
Starting Condition​​: Home Page 
Success​​: Find a ticket that meets the given criteria 
Fail​​: The user is unable to purchase a ticket with for the route with the specific constraints 
Needs​​: Train routes that we know have the necessary amenities for a given situation 
(dogs and bikes are not allowed on all routes)   
Scenario​​:  
You live in Seattle, but need to attend a conference on November 20th in Portland (station PDX). 
After the conference, you need to visit a friend in Vancouver, BC (traveling on the 21st). After your 
visit, you need to return to Seattle on the 22nd. Book this trip as one journey. 

Task 3: Search Feature 

Goal​​: Use the search feature to answer a given question. 

Starting Condition​​: Home Page 
Success​​: Find the information that answers the user’s question 
Fail​​: The user does not discover the answer to their question 
Needs​​: Search terms we know will highlight the issues in the search page 
Scenario​​:  
You are thinking about traveling overnight with a friend on the Coast Starlight train later this 
month. Your friend is gluten free and wants to know if there are meals available for their dietary 
restrictions.  
Mid Task Prompt: “​​Now see if you can find the information another way”  

Task 4: Train Tracking 

Goal​​: Ask a user to track the status of a train and then navigate from the schedule tracking 
system to the map train tracking system 
Starting Condition​​: Home Page 
Success​​: Get between the two trackers and track the same train 
 Fail​​: The user is unable to locate the status of the train or they are unable to navigate 
from the schedule to the map  
Needs​​: A train/bus number to give users 
Scenario​​:  
Your friend from Portland is currently en route to visit you today on a train traveling from the 
Portland Union Station to the Seattle King Street Station. Track the train to see if it will be arriving 
on time.  
Mid Task Prompt​​: “Now see if you can find an alternate method to track a train”  
   
Participant Task Scenarios    
Participant ____ 
Task 1  
You and a friend want to travel to Portland, Oregon on Friday, November 30th for a concert the 
following day. You will be leaving from Seattle, Washington. You would like to find the best ticket. 
 
Scenario A: ​Your friend uses a wheelchair that may be stowed during travel. This friend 
also has a service dog that will assist them during travel. 
 
Scenario B: ​Your friend is a member of the military. Both you and your friend will be 
transporting your bikes with you on your trip. 
 
How difficult was it to complete the task?  
Easy Difficult 
1 ------------- 2 --------------- 3 --------------- 4 ---------------- 5 
How understandable was the process of completing this task? 
Understandable Confusing  
1 ------------- 2 --------------- 3 --------------- 4 ---------------- 5 
Anything else you want to tell us? 
 
  
 
   
 Participant ____ 
Task 2 
You live in Seattle, but need to attend a conference on November 20th in Portland (station PDX). 
After the conference, you need to visit a friend in Vancouver, BC (traveling on the 21st). After your 
visit, you need to return to Seattle on the 22nd. Book this trip as one journey. 
 
How difficult was it to complete the task?  
Easy Difficult 
1 ------------- 2 --------------- 3 --------------- 4 ---------------- 5 
How understandable was the process of completing this task? 
Understandable Confusing  
1 ------------- 2 --------------- 3 --------------- 4 ---------------- 5 
Anything else you want to tell us?  
 
 
 
 
   
 Participant ____ 
Task 3 
You are thinking about traveling overnight with a friend on the Coast Starlight train later this 
month. Your friend is gluten free and wants to know if there are meals available for their dietary 
restrictions.   
 
How difficult was it to complete the task?  
Easy Difficult 
1 ------------- 2 --------------- 3 --------------- 4 ---------------- 5 
How understandable was the process of completing this task? 
Understandable Confusing  
1 ------------- 2 --------------- 3 --------------- 4 ---------------- 5 
Anything else you want to tell us?  
 
 
 
 
   
 Participant ____ 
Task 4 
Your friend from Portland is currently en route to visit you today on a train traveling from the 
Portland Union Station to the Seattle King Street Station. Track the train to see if it will be arriving 
on time.  
 
How difficult was it to complete the task?  
Easy Difficult 
1 ------------- 2 --------------- 3 --------------- 4 ---------------- 5 
 
How understandable was the process of completing this task? 
Understandable Confusing  
1 ------------- 2 --------------- 3 --------------- 4 ---------------- 5 
 
Anything else you want to tell us?    
Data Logging and Note Taking    
Task  Time Stamp  Task Success  Errors  Assists  Non Verbal Response 

1           
 
 
 

2           
 
 
 

3           
 
 
 

4           
 
 
 
 
Task  Positive Comments  Negative Comments  Other 

1       
 
 
 

2       
 
 
 

3       
 
 
 

4       
 
 
 
Other Notes:  
 
___________________________________________________________________________

___________________________________________________________________________

___________________________________________________________________________

___________________________________________________________________________

___________________________________________________________________________

___________________________________________________________________________

___________________________________________________________________________

___________________________________________________________________________

___________________________________________________________________________

___________________________________________________________________________

___________________________________________________________________________

___________________________________________________________________________

___________________________________________________________________________

___________________________________________________________________________

___________________________________________________________________________

___________________________________________________________________________

___________________________________________________________________________

___________________________________________________________________________