Part II.

3. Research Procedure

The task objective is to facilitate the process of collection understanding

through intuitive and fluid interaction. Collection understanding is a
challenging concept. How can an investigator deduce when an individual
has attained an understanding of a collection? In some sense, subjectivity
plays a major role. Defining what constitutes new knowledge or insight is
a highly personal experience dependent upon individual backgrounds and
prior familiarity with Picasso. To gauge a qualitative experience, the study
will employ a think-aloud protocol, in which an active conversation with
the participant provides us with some sense of what thoughts and creative
relationships are formulated in the participant’s mind.

Participants will be given specific task(s) to complete. Participants will

answer one or more of these questions by writing a descriptive response,
in addition to creating a collection of images to support that response. The
tasks can be any one of the following questions:

1. What do you think is Picasso’s main focus/topic/subject in this

collection?
2. Choose images from the collection that you like and dislike.
Compare/contrast these images and state why you liked/disliked
them.
3. Which works of Picasso appealed to you the most? What are the
connections/relationships among them?

Each subject needs to develop some sense or understanding of the

collection in order to complete task(s). A portion of each task will be
directed towards the participant’s subjectivity in interpreting the task and
coming to their own conclusions about the collection.

This test is designed to give the user a qualitative experience interacting

with the images (i.e. the actual contents) in the Picasso collection. How
the user goes about finding out what is in the collection is a qualitative
experience. Quantitative measures of feature use will also be logged to
illustrate whether the task of understanding the collection was tedious.

Approximately 30 participants will be divided equally and randomly. The

first set of 15 participants will use the Collection Understanding interface
(see Figure 1) and the second set (of 15 participants), the “control” group,
will use the IR interface (see Figure 2). Each group will be asked to
provide a representative set of artifacts that exemplify their conclusions.
Using the Collection Understanding tool, users may illustrate the resulting
collage or choose the artifacts in a different window. With the IR interface,
users will right-click on the image, choose the “Save Image As…” menu
option, and place all their images in one folder.
Figure 1: Collection Understanding Tool (Streaming Collage Visualization)

Figure 2: Information Retrieval (IR) interface

 Participants will be given a pre-task questionnaire in order to collect
background information about their prior knowledge about Picasso.
Afterwards, the investigator will provide a brief tutorial to acquaint users
with the respective tool, and the specific task will be administered. The
program will be instrumented to produce logs of user actions. During the
task, participants will engage in a think-aloud protocol and write their
descriptive response for the task. During and upon completion of the task,
(non-intrusive) screenshots will be taken to provide context for the user’s
experience. That is, the investigator will have a permanent record of the
scope of the collection that the participant explored. After the task is
completed, a post-task questionnaire will be issued to find out specific
details about the experience and feedback about the task.

Each participant should complete the tasks including questionnaires and

the tutorial in approximately one hour though no serious time restrictions
will be imposed on the tasks for any of the participants (see Table 1).

Events Estimated Time/minutes

Pre-evaluation questionnaire 5
Tutorial 5
Specific task: (see above) 40
Post-evaluation questionnaire 10

Table 1: Events and Time to complete events

Evaluation Measures

Data collection will consist of logging quantitative data and subjective

perspectives of the participants. These measures are listed below:

Quantitative Measurements
The following data will be logged during/after the experiment:
1. time taken to complete the task
2. number of errors/problems/bugs encountered with the interface
3. the number of different result sets that were generated for each
interface
4. the mechanism used to navigate the collection
i. For the Collection Understanding interface: typing metadata
values vs. using metadata filtering criteria for viewing
different set of artifacts
ii. For the IR interface: number of pages visited when viewing
another set of collection artifacts
5. frequency with which participant selects a detailed view of the image
6. frequency with which user requests metadata values for the images

Qualitative Experience
Qualitative data will consist of:
1. screenshots captured during the experiment
2. participant’s think-aloud thoughts
3. final representative set(s) of artifacts chosen by participant
4. written description of user’s perspective about the tasks
Pre-evaluation Questionnaire

1. What is your age range?

a. 18-25
b. 26-35
c. 36-45
d. 46-55
e. >55

2. Rate your knowledge about Picasso.

a. Novice
b. Intermediate
c. Expert

3. On a scale from 1-5, how much new knowledge do you hope to acquire
about Picasso?

1. no new knowledge
2. little new knowledge
3. medium new knowledge
4. significant insight

4. On a scale from 1-5, rate your interest in learning about Picasso.

1. no interest
2. little interest
3. medium interest
4. great interest

5. Have you ever been to a Picasso exhibit in a museum gallery? (yes/no)

Task Question

1. What do you think is Picasso’s main focus/topic/subject in this

collection (using the images you collected to illustrate your answer)?

___________________________________________________________
___________________________________________________________
___________________________________________________________
___________________________________________________________
___________________________________________________________
___________________________________________________________
___________________________________________________________
___________________________________________________________
___________________________________________________________
___________________________________________________________
___________________________________________________________
___________________________________________________________
___________________________________________________________
___________________________________________________________
___________________________________________________________
___________________________________________________________
___________________________________________________________
___________________________________________________________
___________________________________________________________
___________________________________________________________
___________________________________________________________
___________________________________________________________
Post-evaluation Questionnaire

1. Did you feel that you had control defining the content of the collection
presented to you (yes/no)?

Please rate your level of control.

1) None; had no control
2) poor
3) o.k.
4) good
5) excellent

Do you think the interface provided a static or dynamic view of

Picasso’s paintings?

2. Did you have the skills you needed to succeed in the activity?

a. not at all
b. a little
c. medium – somewhat
d. substantially
e. very well

3. How easy was it for you to navigate your way around the entire
collection?
1. very difficult
2. moderately difficult
3. neither easy nor difficult
4. moderately easy
5. very easy

4. How flexible was the interface in allowing you to formulate your own
collection search criteria?

1. very flexible
2. moderately flexible
3. not flexible

5. What insights about Picasso’s work did you have, if any? That is, what
interesting knowledge did you learn about Picasso that you didn’t
previously know?

___________________________________________________________
___________________________________________________________
___________________________________________________________
___________________________________________________________
___________________________________________________________
___________________________________________________________

6. On a scale of 1-5, rate your enjoyment learning more about Picasso in

the interface you used.
1) did not enjoy the experience at all
2) minimal enjoyment
3) neutral
4) medium enjoyment
5) very enjoyable

7. What aspects of the interface made the experience

enjoyable/frustrating/fulfilling?

___________________________________________________________
___________________________________________________________
___________________________________________________________
___________________________________________________________
___________________________________________________________
___________________________________________________________

8. Did you find the process of viewing the images tedious?

1. very tedious
2. moderately tedious
3. not tedious

9. Please rate your progress in viewing and understanding the entire

collection.
1. a great deal of progress
2. moderate progress
3. a little progress
4. no progress

10. Was the activity challenging for you?

a. very low
b. low
c. medium
d. substantially
e. extremely
11. Did the interface provide meaningful feedback to help you to complete
your task?
1. a great deal of feedback
2. moderate feedback
3. a little feedback
4. no feedback

How did the interface provide meaningful feedback, if any?

___________________________________________________________
___________________________________________________________
___________________________________________________________
___________________________________________________________
___________________________________________________________

12. Was it difficult for you to discover new Picasso artifacts that you
wanted to learn about?

1. very difficult
2. moderately difficult
3. neither easy nor difficult
4. moderately easy
5. very easy

13. Did you re-define your goals based on the results you received? If so,
how?

___________________________________________________________
___________________________________________________________
___________________________________________________________
___________________________________________________________
 INFORMATION SHEET
Collection Understanding of Picasso

You have been asked to participate in a research study that helps you to understand the Picasso
collection. You were selected to be a possible participant because of your interest to know more
about Picasso. A total of approximately 30 people have been asked to participate in this study.
The purpose of this study is to determine whether the interface provided helps you to easily and
effortlessly learn about Picasso and his works. The results of this study will be used for a CPSC
610 Hypertext class project.

If you agree to this study, you will be asked to complete one task - think aloud your thoughts
while performing the task and also write a descriptive response about the task. You will also fill
out pre- and post-task questionnaires about your experience with the Picasso collection. You
agree to allow the investigator to monitor your interaction with the software during the
experiment. This study will only take about one hour. The only risk associated with this study is
your loss of time. The benefit of participation is that you will have a greater appreciation for
Picasso’s work. You will receive no monetary compensation for this study.

This study is anonymous. Your interaction with the interface is assigned an identification number
(ID) only to distinguish your responses from another participant’s responses. This ID is in no way
associated with your name. The records of this study will be kept private. No identifiers linking
you to the study will be included in any sort of report that might be published. Research records
will be stored securely and only the investigator, Michelle Chang, and the course professor, Dr.
Andruid Kerne, will have access to the records.

Your decision whether or not to participate will not affect your current or future relations with
Texas A&M University. If you decide to participate, you are free to refuse to answer any of the
questions in the questionnaire that make you uncomfortable. You can withdraw at any time
without your relations with the university, job, benefits, etc., being affected. You can contact
Michelle Chang at 979-845-4924 or e-mail mchang@tamu.edu with any questions about this
study. Additionally, the investigator can terminate your participation if you are unwilling to
complete the given task.

The Institutional Review Board – Human Subjects in Research, Texas A&M University, has
reviewed this research study. For research-related problems or questions regarding subjects’
rights, you can contact the Institutional Review Board through Dr. Michael W. Buckley, Director
of Research Compliance, Office of Vice President for Research at 979-845-8585
(mwbuckley@tamu.edu).

You may ask any questions that you may have at this point. You will be given a copy of this
information sheet for your records. By completing the task given, you agree to participate in this
study. Thank you for your time.

For further information, contact:

Michelle Chang (Principal Investigator) Dr. Andruid Kerne (Course professor)
408G H.R. Bright Building, 402A H. R. Bright Building
Texas A&M University, Texas A&M University
College Station, TX 77843. College Station, TX 77843-3112
Phone: (979) 845-4924 Phone: (979) 862-3217
E-mail: mchang@tamu.edu E-mail: andruid@cs.tamu.edu