1

Macroinvertebrates.org Usability Study

With Trainers and Citizen Scientists
Study Description
With National Science Foundation funding the Learning Media Design Center at Carnegie
Mellon University in partnership with the University of Pittsburgh Center for Learning in Out-ofSchool Environments, Carnegie Museum of Natural History, Stroud Water Research Center,
and Clemson University are developing a new kind of open educational resource, an online
digital teaching collection of aquatic macroinvertebrates <www.macroinvertebrates.org> to
support stream insect identification tasks in education and citizen science biomonitoring
activities. The underlying goal for this project is to make the process of learning to identify
aquatic macroinvertebrates easier for non-experts by increasing their confidence, engagement
and accuracy in taxonomic identification tasks, and thereby improving the quality of
biomonitoring data that groups can contribute to citizen science research and environmental
decision-making. This study will be conducted over three years as we evaluate the initial
Demonstration project, Prototype 1 (alpha), Prototype 2 (beta) and the final release of the
website/application with trainers and volunteers involved in water quality biomonitoring
activities.
Invitation to participate in research:
You are invited to take part in this research study because of your professional work or
volunteer participation water quality biomonitoring activities. This study is being conducted with
40 participants over three year using a verbal think aloud site review with semi-structured
questions and prompts, and a subsequent online survey.
Research Activities:
If you decide to take part in this research study, you will be asked to participate in a website
review session using a think-aloud method where your interactions with the website under study
(www.macroinvertebrates.org) will be recorded as you talk about it. Please note, your screen
actions, audio and video during the interview will be recorded. In addition, you will be asked to
complete a brief online survey. Survey questions will be limited in number to 10 questions or
less, and include scale, frequency, as well as open field text responses about your impressions
of the site, and your opinions about its value for biomonitoring training purposes. The thinkaloud protocol and survey is designed to be completed within a 30 minute time period, and will
include questions about your training and volunteer biomonitoring experiences, and taxonomic
identification practices. We will link the survey data and interview responses.
Participation in this research study
Your participation in this research study is completely voluntary, and you may withdraw from
this research study at any time. If you should wish to stop participating in the study at any time
throughout the year, you should alert the researcher who is interviewing you.
You could be withdrawn from the study if you become unable or unwilling to follow the
instructions given by the investigators. Early withdrawal will have no effect on your relationship
with the Carnegie Mellon University.

Possible risks/benefits/costs of participation

With this study, there is a possibility of a breach of confidentiality. However, the research team
has been trained to avoid this, and all data will be kept in a locked file cabinet. Beyond the
possibility of a breach of confidentiality, no adults will be engaged in any activities that pose
dangers or risks other than those that normally occur during working hours and in professional
interactions. You will not be charged, nor will you receive compensation for any of the activities
performed for the purpose of this research study. There are no direct benefits to those
participating in this study.
Confidentiality
By participating in this research, you understand and agree that Carnegie Mellon may be required
to disclose your consent form, data and other personally identifiable information as required by
law, regulation, subpoena or court order. Otherwise, your confidentiality will be maintained in the
following manner:
Your data and consent form will be kept separate. Your consent form will be stored in a secure
location on Carnegie Mellon property and will not be disclosed to third parties. Sharing of data
with other researchers will only be done in such a manner that you will not be identified.
By participating, you understand and agree that the data and information gathered during this
study may be used by Carnegie Mellon and published and/or disclosed by Carnegie Mellon to
others outside of Carnegie Mellon. Video/screencast data will be used exclusively for research
purposes and stored in a secure location. However, your name, address, contact information and
other direct personal identifiers will not be mentioned in any such publication or dissemination of
the research data and/or results by Carnegie Mellon. Note that per regulation all research data
must be kept for a minimum of 3 years.
Right to Ask Questions & Contact Information
If you have any questions about this study, you should feel free to ask them by contacting the
Principal Investigator. If you have questions later, desire additional information, or wish to
withdraw your participation please contact the Principal Investigator by mail, phone or e-mail in
accordance with the contact information listed below.
Principal Investigator:

Other Investigator(s):

Sponsor(s):

Marti Louw, Director

Learning Media Design Center
Human Computer Interaction Institute
Carnegie Mellon University
5000 Forbes Ave.
tel: 412-268-6917 email: martil@cmu.edu
Ahmed Ansari, Lauren Allen,
Learning Media Design Center
NSF #1516149

3
Learning to See, Seeing to Learn:
A Sociotechnical System Supporting Taxonomic Identification
Activities in Volunteer-Based Water Quality Biomonitoring

If you have questions pertaining to your rights as a research participant; or to report concerns to
this study, you should contact the Office of Research integrity and Compliance at Carnegie
Mellon University. Email: irb-review@andrew.cmu.edu Phone: 412-268-1901 or 412-268-5460.

Macroinvertebrates.org Usability Review Study

Think Aloud Protocol & Follow-up Survey
[ Protocol Description ]
The Macroinvertebrates.org Usability Studies will be conducted in two parts:
Part 1 - Remote think-aloud site review and semi-structured interview (15 - 20 minutes)
Part 2 - Online usability survey (10 minutes; 8 questions)
Recruitment target: 10 participants (5 trainers + 5 citizen scientists/learners) per prototype
version (demonstration site, prototype 1 (alpha), prototype 2 (beta), final release).
Trainers will be recruited through our advisory panel contacts, our co-investigators professional
networks, and list-serv sites for volunteer biomonitoring organizations. Adult citizen science
volunteers will be recruited through our partner volunteer biomonitoring organizations
(Pittsburgh Parks Conservancys Young Naturalists, Trout Unlimiteds Coldwater Conservation
Corps, Maryland DNRs Streamwaders, Nature Abounds Sr. Environmental Corps, and
Dickinson Colleges ALLARM). We will use a snowball networking method to increase
recruitment as needed.
In Part 1, may be conducted as either a remote think-aloud protocol, where participants will be
invited to join a video conference call and asked to share their screen with the website
www.macroinvertebrates.org loaded, or conducted locally with the researcher using a screen
capture software. In both cases, the researchers will request verbal assent for permission to
record a video/screencast of the user study session. If participants agree, they will be instructed
to begin by freely explore the website, and as they do so asked to provide a free running
narrative of their impressions, describing out-loud what they are thinking, doing and feeling as
they navigate the site. Researchers will uses prompts like: please keep talking, what are you
doing/looking for now, what do you think that is for to keep the commentary going. The user
will then be asked to describe how they might use this site to support a taxonomic identification
tasks, and how might they use this for training citizen science volunteers learning to do
taxonomic identification work, or using it as a learning aid. Once the user has fully explore the
site, and responded to these questions, the researchers will close the video conference and the

4
audiovisual screencast recording will be locally stored to the investigator's computer and saved
to box using a naming convention ties relates Part 1 interview to Part 2 survey data without
recording individual names. Participants will then be asked to respond to an emailed survey.
Read Script, OK to record ...
Opening Questions:
1. Professional Background in regards to water quality biomonitoring and training
2. How would you characterize your taxonomic expertise
3. Tell us a little bit about the kinds of training you do with volunteers,
4. What tools materials, resources Streamside vs lab; family / order
5. Whats hard easy about training volunteers?
6. Volunteer motivations, why coming?
Request to open browser:
What browser do you use? (suggest Chrome/Firefox). Now take a few moments to look at site.
Is this your first time at the site?
Give us your impressions? Remember the user is always right!
How might you use a site like this for volunteer training?
If you were going to describe this site to a colleague what would you say, and what would you
recommend it would be good for?
Are their particular groups or purposes this site would be good for.

In Part 2, participants will receive an email with an online linked and/or an attached PDF version
of the Macroinvertebrates.org Usefulness & Usability Survey.

***************ONLINE USABILITY SURVEY**********************

With funding from the National Science Foundation, we are planning to improve the
www.macroinvertebrates.org, an online collection of aquatic macroinvertebrates. After you have
taken some time to explore this site, wed like to ask you to complete the following survey to
help us understand how useful and usable the website is, and for whom. Your responses will
help us determine what steps we should take to develop this demonstration project into a fullscale online collection.
Would you be willing to participate in this study? [ Yes ] [ No ]

5
1. Do you feel that level of information on the website is:
0

Not enough detail

Just Right

Too
detailed

2. In terms of navigation, do you feel that the website is:

0

Not at all easy to

navigate

Just right

Very easy to
navigate

3. Are you able to easily find the diagnostic characters you are looking for on the website?
0

Not at all

Somewhat easily

Very easily

4. Do you think the website does a good job of providing additional supporting content (habitat
information, characteristics, videos etc.)?
0

Not at all

Somewhat
good

Very well

6
5a. How useful would this site be to you in your work?
0

Not Useful

Somewhat useful

Very Useful

5b. Please explain your response in terms of the usefulness of the site to you in your work.

6. Considering the level of content on this website and its overall design, how would you rate this as
a learning or training resource for following types of users:
[Not appropriate, Somewhat appropriate, Appropriate, Very Appropriate ]

a.
b.
c.
d.
e.
f.
g.
h.
i.
j.

General Public__________________________________
Trainers with an average level of taxonomic expertise_________________________________
Trainers with a high level of taxonomic expertise
Novice volunteers__________________________________
Semi-experienced volunteers__________________________________
Experienced volunteers__________________________________
School teachers__________________________________
Entomology students__________________________________
Professional Taxonomists/Entomologists__________________________________
Other (Please specify): __________________________________

7a. How likely are you to recommend this site to a colleague?

0

Not at all likely

Somewhat likely

Extremely likely

7b. How would you describe this site to a colleague?

8. What changes or additions would you like to see made to www.macroinvertebrates.org?

7
Summary of Activities
Procedure

Time

Video-call and remote screen casting setup (Invitation sent to participant to join
session)

3 min

Remote think-aloud protocol

7 min

Interview questions

10 min

Online Survey

10 min

[ Recruitment Email ]
Dear _______,
With National Science Foundation funding the Learning Media Design Center at Carnegie
Mellon University in partnership with the University of Pittsburgh Center for Learning in Out-ofSchool Environments, Carnegie Museum of Natural History, Stroud Water Research Center,
and Clemson University are developing a new kind of open educational resource, an online
digital teaching collection of aquatic macroinvertebrates <www.macroinvertebrates.org> to
support stream insect identification tasks in education and citizen science biomonitoring
activities. The underlying goal for this project is to make the process of learning to identify
aquatic macroinvertebrates easier for non-experts by increasing their confidence, engagement
and accuracy in taxonomic identification tasks, and thereby improving the quality of
biomonitoring data that groups can contribute to citizen science research and environmental
decision-making.
For this study, we would like to ask you to participate in an online review session of the website
macroinvertebrates.org via a videoconference interview and screencast capture, followed by a
short online survey. The interview and survey that should take no longer than 30 minutes of
your time. Your answers will remain confidential.
Please let us know if you would be willing to participate in this study. If so we will contact you by
email to find a convenient time to schedule an online review session and interview.
Thank you for considering this request.
Yours Sincerely,
Marti Louw, Director
Learning Media Design Center
Carnegie Mellon University
martil@cmu.edu tel: 412-268-6917