Chapter 4 (Smith and Davis) : Non-Experimental Methods

Chapter 4 (Smith and Davis)
Non-Experimental Methods
An Overview Of This Chapter
 Let’s Get Back to Methods again
In this chapter, I want to jump back into a “methods” orientation
rather than discussing statistical calculations, as we’ve focused
on for the last few chapters
– Well … I will show you how to calculate one statistic (inter-
rater reliability) in Part One, but it will be brief
This chapter is focused more on setting up stimulus material
and collecting data. Since your labs focus on study materials,
this is the perfect time to look at developing good survey and
questionnaire material your study
An Overview Of This Chapter
 In Chapter four, we cover the following:
Part One – Descriptive Methods
Part Two – Ex-Post Facto Research
Part Three – Surveys, Questionnaires, Tests, and Inventories
Part Four – Sampling Considerations and Research Strategies
Part Five – Basic Research Strategies
Part Six – An Eye Toward The Future
Part One
Descriptive Methods
Descriptive Methods
 Descriptive methods focus on non-experimental methods that do
not involve the manipulation of any variable by the experimenter
Since we don’t manipulate variables, the descriptive methods
in this section focus on research that are not causal in nature
In this section, you will learn about …

– 1). Archival research
– 2). Naturalistic observations
– 3). Behavioral recording techniques
– 4). Reliability of behavioral recording techniques
Descriptive Methods
 1). Archival Research refers to non-experimental strategies that
involve studying existing records, including (but not limited to) …
Historical Accounts Military Data

Census Data Business Records
Court Records Census Data
Police Crime Reports Marriage Records
Published Research Articles Divorce Records
School Records Driving Records
Political Speeches Hospital Records
Descriptive Methods
 1). Archival Research
Researchers who conduct archival research usually go into
archival research with specific research questions in mind
– For example, they might want to look at divorce records
after terrorist attacks to see if rates rise or fall (researchers
saw divorces drop after the 1995 Oklahoma City bombing
case, probably because people come together after attacks)
Yet gaining access to archival data can be tricky

– Some resources may be incomplete, and thus they might
need to be supplemented with other measures
Descriptive Methods
 1). Archival Research – An example
Baker (1989) looked at various records to track the rates of
psychopathology in both the adoptive and biological families of
severely disturbed adults who were adopted as infants
– Baker carried out his studies in Denmark, where archival
records were more accurate, more complete, and easier to
obtain than similar records in other countries
Of course, the source of the information in archival studies

might be somewhat biased. Let’s play ball …
Descriptive Methods
Tannenbaum and Noah and the “home
team” in local newspapers.
– Consider the headlines below. Where
do you think each was published?
– “Yankees Edge out Red Sox 8-7”
– “Yankees Bludgeon Red Sox 8-7”

Descriptive Methods
– Consider the headlines below. Where
do you think each was published?
– “Yankees Edge out Red Sox 8-7”

Boston headline!
– “Yankees Bludgeon Red Sox 8-7”
New York City headline!
Descriptive Methods
– Newspapers root for the home team,
so they describe losses “not too bad”
and wins ass “huge!”
– There’s a lot of information in the world
for us to look at, but we must recognize
that the archival data might be biased,
incomplete, or just plain wrong
– Why? Well …
Descriptive Methods
 1). Archival Research – Potential problems in archival research
First, you may not know who left the data you are investigating
– Was it a careful, focused researcher or a conspiracy theorist
Second, the original author may have been selective in what

they chose to write (an author may misrepresent other’s ideas)
– FOX News vs. CNN vs. MSNBC: They all believe the exact
same thing, right? (Ha!)
Descriptive Methods
 1). Archival Research – Potential problems in archival research
Third, not all data may be in a format that YOU need
– Archival data you find on murder may have broad categories
like “shooting”, “knifing”, or even “bludgeoning”, but not the
“murder by anvil” dimension YOU may really want to study!
Descriptive Methods
 2). Naturalistic Observations are purely observational:
We observe subjects in their own natural environment without
making any attempt to control or manipulate variables, making
such observations (once again) non-experimental
The goals of naturalistic observations are to

– A. Describe the behavior as it occurs in the natural setting
(without the artificiality of the laboratory)
– B. Describe the variables that are present and relationships
among the variables
Descriptive Methods
Of course, there are problems with naturalistic observations
– A). Reactance / Reactivity Effect (or The Hawthorne Effect):
Participants may give biased or altered responses simply
because they know they are being observed
Descriptive Methods
Of course, there are problems with naturalistic observations
– B). Getting access to the environment might be tough. There
might be gatekeepers preventing your access.
It may be tough to stroll into a kindergarten classroom to
observe children learning without permission!
– C). Being objective can be a hurdle as well.

Descriptive Methods
How do we deal with potential bias in observational designs?
– Use well-trained researchers to observe behaviors
Training may involve watching for specific events, but
researchers need to know what those specific “events”
are before observation begins
Consider some behavioral recording techniques …
Descriptive Methods
 3). Behavioral Recording Techniques
Behavioral categories (coding schemes) include general and
specific classes of behavior of interest to the researcher
– Defining behavioral categories can be difficult
For example, how do you define punishment?
Descriptive Methods
 3). Behavioral Recording Techniques – Getting category ideas
There are dozens of ways that we can decide what behaviors
to record. Consider the behavioral category “aggression”
– First, we can make informal, preliminary observations of
potential “subjects” (essentially a pilot observation session)
We look at “aggressive” behaviors on the playground
and develop categories based on what we see
Descriptive Methods
 3). Behavioral Recording Techniques – Getting category ideas
There are dozens of ways that we can decide what behaviors
to record. Consider the behavioral category “aggression”
– Second, we can conduct a literature review to get basic
ideas about what others have done. We can look at their
categories and adapt/adjust them to suit our own study
– Third (and related), we can ask experts about the best

categories to use
Descriptive Methods
Once you have your categories, you observe! There are lots of
ways to do this …
– A. Frequency method
– B. Duration method
– C. Intervals method
– D. Time sampling method
– E. Individual sampling method
– F. Situation / event sampling method
– G. Recording methods for A through F above
Descriptive Methods
A. The frequency method records the number of times that a
particular behavior occurs within a set time period
Descriptive Methods
B. The duration method records how long a specific behavior
lasts (that is, from start to finish, how long does an event last?)
Descriptive Methods
C. The intervals method divides the observation period into
discrete time intervals and records whether or not a behavior
occurs in that time period
Descriptive Methods
The frequency, duration, and intervals methods are all useful
for counting the frequency and length of behavior, but keep in
mind that behavior is complex, and the people we observe can
do things that are not easily categorized
Descriptive Methods
Also, think about that intervals method again. Can you spot the
big problem with this method?
– Let’s say you are watching a worm-eating event live and
taking lots of notes. What happens when you look down at
your notes to record your observations? What might you be
missing while distracted with your notes?
Descriptive Methods
D. In time sampling, the researcher scans the event for some
given time period and then records the behavior during the next
time period (observes, then records, observes, records, etc.)
– A benefit is that you have some control over when you look
and don’t look. You will miss stuff, but in a controlled way
Descriptive Methods
E. In individual sampling, you select a single subject and then
record his or her individual behavior across a time span
Descriptive Methods
F. In situation (or event) sampling, you observe one behavior
but record all instances of that behavior in different situations
Descriptive Methods
G. Making permanent recordings of observations aids coding
– You can watch the behavior multiple times if needed
– You can have multiple observers watch the behavior
– You can hide a camera more easily than hiding an observer
Descriptive Methods
G. Making permanent recordings of observations aids coding
– Paper / pencil coding forms can be used to record
– Tape recorders
– Video recorders
Descriptive Methods
 3). Behavioral Recording Techniques – To sum up!
A. Frequency method – How often does it occur?
B. Duration method – How long does it occur?
C. Interval method – Does it occur in a given time frame?
D. Time sampling – Watch then record; repeat as needed
E. Individual sampling – Watch only one individual
F. Situation / event sampling – Watch for a specific event
Descriptive Methods
 4). Reliability of behavioral recording techniques
As I mentioned, observer training is really important before
observations begin, and this is true. The reason is that people
(and I consider researchers among them!) can be biased
– We like to think of ourselves as objective, but researchers
make predictions about what we think will occur. We might
be vigilant when looking for occurrences that confirm our
predictions, but inclined to ignore (or just possibly) things
that disconfirm our predictions.
– So how can we see if researchers are biased in their
observations?
Descriptive Methods
Inter-observer (or inter-rater) reliability provides an empirical
index for measuring agreement among observers
– Reliability helps insure observer accuracy, aiding replication
You see if multiple observers meet established research
standards by agreeing with each other’s observations
If you detect differences, you can retrain the observers
But how do you find inter-rater reliability? Consider some data:

Think about this % Agreement
Time Period Psychologist #1 Psychologist #2
1 0 0
2 0 0
3 0 0
4 1 0
5 0 0
6 0 0
7 0 1
8 0 0
9 0 0
10 0 0
Two psychologists rate aggressive actions by kids in 10 time periods
1 0 0
2 0 0
3 0 0
4 1 0
5 0 0
6 0 0
7 0 1
8 0 0
9 0 0
10 0 0
Eight out of ten times, the psychologists agree! 80% agreement. Yay!
Descriptive Methods
I bet you already know how to calculate percent agreement. In
8 observations out of 10, the observers found the same thing
Percent agreement is an estimate: It can easily go up or down

– If “rater agreements” must be exact matches (100% agree),
percent agreement can underestimate inter-rater agreement
Descriptive Methods
– Let’s say our two psychologists look for aggression in terms
of punching. Both see face-punching as aggressive, but one
categorizes arm-punching as playful while the other sees it
as aggressive. This disagreement can lower percent scores,
calling for better training on WHAT qualifies as aggressive
Descriptive Methods
– Yet behavior very high or very low in frequency influences
agreement as well, and may lead to the overestimation of
inter-rater agreement
– Recall the % agreement among our two psychologists …

1 0 0
2 0 0
3 0 0
4 1 0
5 0 0
6 0 0
7 0 1
8 0 0
9 0 0
10 0 0
The times when there is no aggression is VERY HIGH in frequency,
and the two psychologists agree 100% of the time it is absent
1 0 0
2 0 0
3 0 0
4 1 0
5 0 0
6 0 0
7 0 1
8 0 0
9 0 0
10 0 0
Yet a big problem in this example is that the two psychologists
NEVER agree when aggression is present (VERY LOW)
Descriptive Methods
Given the unreliable (ha!) nature of the percentage score, there
are several statistical ways that psychologists can assess inter-
rater reliability, including …
– A. Cohen’s Kappa (assesses agreement relative to chance)
– B. Interclass Correlation Coefficient (focuses on group data)
– C. Pearson’s Product Moment Correlation (often denoted by
the letter r, this ranges from –1 to +1)
– We will discuss the calculations for these a bit more in our

next Chapter (Chapter 6, Salkind). Not stats for now!
Descriptive Methods
 4). Reliability of behavioral recording techniques – Pearson’s r
A brief word about the Pearson Product-Moment Correlation, a
coefficient that measures the strength of a linear association
between two variables.
– It attempts to draw a line of best fit through the data of two
variables, and r indicates how far away data points are to
this line of best fit (how well the data points fit this new
model/line of best fit).
– I want to spend more time on Pearson … but not until we
get to 16 (Salkind)! Read up on it if you like, but let’s move
onto …
Part Two
Ex Post Facto Studies
 Ex post facto studies refer to study designs in which the variables
to be studied are selected after they have already occurred
That is, you use the IV after the fact (it varied before you began
the study). This is also called a quasi-experimental study.
– Gender, place of employment, etc. have already occurred,
so you cannot manipulate these variables, but you can still
look at them in a research study
– For example …
… you might observe that students from one town have higher
grades than students from a different town, even though both
attend the same high school.
– Would "being from a certain town" explain the differences?
– In ex post facto studies, researchers can explore specific
reasons for the differences, such as variation in income,
ethnicity, parental support, etc. (all variables beyond the
control of the experimenter), but …
…as with other non-experimental research in this chapter, ex
post facto studies cannot lead to "cause and effect" conclusions
– All we can say is there is a relationship between variables.
“Students from Town A have higher grades, which might
be attributable to the higher educational level of who
decided to live in Town A”
Possible, but only one potential explanation!
Part Three
Surveys, Questionnaires,
Tests, and Inventories
Surveys and Questionnaires
 In Part three of this chapter, we will discuss several elements of
survey research, focusing on
1). Steps in the survey process
2). Types of surveys and questionnaires
3). Developing a good survey or questionnaire
– This section will be particularly important as you start to
design questions for your second study
 1). In a nutshell, here are the steps in survey research
Determine what area of information we seek
Define the population we will study
Decide how we will administer the survey
Construct the first draft of the survey instrument; edit & refine
Pretest the survey with a pilot sample; refine it further
Develop a sampling frame and draw a representative sample
Administer the final form of the instrument to the final sample
Analyze, interpret, and communicate the results
 2). Types of surveys and questionnaires
There are several types of surveys researchers may use …
– A. Descriptive surveys
– B. Analytic surveys
– C. Pilot test surveys
– D. Mail surveys
– E. Internet surveys
– F. Telephone surveys
– G. Group administered surveys
– H. Personal face-to-face surveys
A. Descriptive surveys seek to determine the percentage of the
population that has a certain characteristic, holds a particular
opinion, or engages in a particular behavior
– Some descriptive surveys may manipulate independent
variables, but others may be purely correlational
– For example …
A. Descriptive surveys
– Surveys can evaluate attitudes
Political opinions
Financial opinions
Media opinions
Worldview opinions
Attitudes toward criminal defendants (human or robot!)
– Surveys can also try to predict behavior
B. Analytic surveys seek to determine what relevant variables
are and how they might be related
– Do you ever “self-diagnose” your illnesses using websites
like WebMD? Bad idea (you’ll come away thinking you are
on death’s door!)
– Well …
B. Analytic surveys seek to determine what relevant variables
are and how they might be related
– Kienhues and Bromme (2012) used an online survey to
measure layperson’s beliefs about medical knowledge and
whether they seek medical help or self-diagnose on
websites like WebMD.
They found that five factors influence lay opinions (such
as certainty of medical knowledge and credibility of
medical information on the internet). Differing levels of
such beliefs might increase patients’ medical compliance
C. Pilot testing involves preliminary, exploratory testing that is
done prior to the complete research project
– Such pilot tests can help researchers know that they are
manipulating variables in the correct way. For example, put
the people below in order of attractiveness (most to least)
D. Mail surveys, obviously, are mailed directly to participants
– Researchers doing mail surveys must be aware of the non-
response bias, a problem that occurs if a large proportion of
participants fail to complete and return your questionnaire
Researchers are thus faced with the question of whether
those who returned the survey differ from those who did
not return the surveys
Maybe “returnees” are elderly, bored, or
helpful participants – yet these may not
be “typical” people from the population
D. Mail surveys, obviously, are mailed directly to participants
– Combating the non-response bias can be achieved by …
1. Having multiple contacts with the respondents
Send a pre-letter alerting participants to the survey
Send a second questionnaire if there is no response
Send a thank you post card
2. Giving potential respondents token gifts as a thank-
you for doing the survey (not contingent on completion)
Thus guilt them into a response!
E. Internet surveys are similar to mail surveys except data is
collected … over the internet, which has a lot of great benefits:
– This technique is best for short, simple surveys
– Computer programs now let you get a little more complex
– You can reach a large sample quickly
– But your sample may not be representative

of your population (you only get computer-
literate participants)
F. Telephone surveys are similar to mail and internet surveys
– Either a live-questioner contacts participants or researchers
can use an interactive voice response (IVR) system (to the
anger of those trying to eat dinner, right!)
Researchers must be careful, though, because “junk” or
“nuisance” calls are against some state laws
G. Group administered surveys are also self-explanatory, with
the survey administered to a large group at the same time
– Disadvantages do crop up with these surveys
Participants may not treat the questionnaire seriously if
they fill it out in a group setting (versus alone)
You may not be able to ensure participant anonymity
Participants may feel pressure to participate in a group,
which has some ethical implications
H. Personal face-to-face interviews involve talking to research
participants directly. This involves different types of interviews
– 1. In structured interviews, the researcher asks participants
prepared questions
This is good, because it eliminates data fluctuations
True / false, multiple choice, short answer, etc.
It is easy to summarize and score responses
Yet these interviews are also inflexible. You may miss
important information as the questions are already set
– 2. In unstructured interviews, the researcher has a general
idea about the issues to be covered, but no set questions
This allows researchers to collect more complete info
and ask different questions that are context-specific
Unfortunately, unstructured responses are difficult to
code and might be very time consuming to collect
– 3. Researchers often use both structures in their research,
combining the two
Ask structured questions first, and then allow participants
to follow up those answers in unstructured interviews
We have talked about some of the disadvantages of research
before. Remember these problems?
– Experimenter biases
– Demand characteristics
– Social desirability problems
Surveys also have these problems. How do we combat them?
– Use trained researchers who ask questions and record
answers in an objective manner
– Use a pilot study to validate the measures before the study
 3). Developing a good survey or questionnaire
We finally get to the part of this chapter that is really relevant to
your second study in Methods Two
Building survey questions …

What are the elements of good questions on a survey?
– A. Operational definitions
– B. Reasonable length
– C. Avoid confusing phrases
– D. Avoid double-barreled questions
– E. Avoid double-negatives
– F. Minimize biased wording
– G. Minimize fence-sitting and floaters
A. Operational definitions clearly define elements of your study
– If you don’t adequately define what you are studying, any
questions you develop might not matter (they are not valid)
– Operational definitions help keep your survey focused
– If I asked you the question, “How successful are you?”, what

do I mean by success? Does it entail:
Wealth? Job power? Happy family life? Good grades?
Without clear definitions, questions are … questionable!
B. Keep your survey to a reasonable length
– Too many questions overwhelms (and bores) participants
– Too few questions doesn’t give you enough information
– Too many powerpoint slides are tiresome (but I hope you
are still enthralled with this course despite the long slides!)
C. Avoid confusing phrases
– The moon question
Does a yes mean you favor colonies only, engineering
only, or that you favor both colonies and engineering?
Does a yes mean you favor manned projects only,
unmanned only, or that you favor both?
What about a no?
– Answers to a confusing questions like provide no useful info.
It has other problems, too, as did a Nixon Gallop question …
D. Avoid double-barreled questions
– A Gallop poll once asked, “Do you think President Nixon
should be impeached and asked to leave the presidency?”
One third of the respondents said yes
– Gallop then asked, “Do you think there is enough evidence
to bring Nixon to trial in front of the senate?”
One half of the respondents said yes
– Notice how the first question actually asks two questions?
Avoid confusing phrases and double-barreled questions!
E. Avoid double-negatives
– Use a simple, direct approach (hopefully no more than 20
words) that avoids double-negatives
– Bad example:
“Do you disagree that rich people are not very happy?”
– Good example:
“To what extent do you think rich people are happy?”
F. Minimize the risk of bias – Which question below is better?
– 1. “If you could choose, would you rather do the kind of work
that people do in jobs outside the home or the kind of work
that people do around the home?”
– 2. “If you could choose, would you rather do the kind of work
that people do in jobs outside the home or the kind of work
that women traditionally have done around the home?”
– Do you think men would have responded that they would

like to work around the house if given Option 2?
G. Minimize fence-sitting and floating
– 1. Fence-sitters are those who see themselves as neutral.
They often choose the middle of the scale and “sit” there
Consider an odd numbered scale (1 to 9). A floater
fence-sitter may always stick in the middle (around 5)
1 2 3 4 5 6 7 8 9
Not at all confident Neutral Very confident
– 1. Fence-sitters are those who see themselves as neutral.
They often choose the middle of the scale and “sit” there
Now consider an even numbered scale (1 to 8). A floater
now has to choose a side (a lower 4 or a higher 5)
1 2 3 4 5 6 7 8
Not at all confident Very confident
– 2. Floaters often choose an answer even if they do not know
how to respond (they may choose a political candidate even
if they don’t know his or her viewpoints)
You may want to offer responders a “do not know” option
so they are not forced to choose an answer at random
Questionnaires in the Real World
Imagine you are a trial consultant and you
are asked to design a questionnaire
For the next few slides, imagine an attorney contacts you and
he wants your help in the trial of a man accused of murdering
his wife and her lover.
– He thinks there is too much negative pretrial publicity in local
papers attacking his client, making it impossible to give the
man a fair trial.
– You decide to help him (for a lot of money of course!) to see
whether his concern is valid. If they is valid, he can submit a
change of venue appeal and move the trial elsewhere
– So, how do you do this? What questions should you ask?
As we talk about our “trial consulting questionnaire”, we will
focus on the following …
– A. Types of questions
– B. The format of the questions
– C. Assembling the questionnaire
– D. Administering your questionnaire
– E. Characteristics of tests and inventories
– F. Types of tests and inventories
– G. Ethics in survey research
A. Types of questions
– Some questions assess participant characteristics
Demographic information (factual participant information,
such as age, gender, race, etc.)
Non-demographic information (like general participant
opinions, attitudes, knowledge, etc.)
– Other questions assess the behaviors of interest (DV’s)
Voter-related surveys need to assess voting-preferences
Surveys assessing prejudice need prejudice-based DV’s
Surveys and Questionnaires – Trials
 3). Developing a good survey or questionnaire – Trial consultant
A. Types of questions – Trial Consulting
– Participant characteristics
Demographics: Sex, age, socio-economic factors
Non-demographics: General attitudes about defendants,
the legal system, the police, etc.
– Behavioral variables for this specific trial
Do potential jurors think this defendant is guilty before
they even hear any case facts?
How aware is the jury pool about the case facts?
B. The format of the question – Open-ended questions
– Open-ended questions allow participants to respond in their
own words, but they have advantages and disadvantages
Advantages: They can provide more information about
the participant opinions that are more accurate and
complete than responses to basic yes-no or multiple
choice questions
Disadvantages: Participants may misunderstand or give
a wrong or incomplete response to questions. You also
need to content analyze responses, which takes time
 3). Developing a good survey or questionnaire – Trial consultant
B. The format of the question – Open-ended questions
– Describe some of the articles you have read or TV programs
you have seen about this case and/or this defendant ______
– In what way do you think that information impacted your
assessment of the defendant’s likely guilt? ______________
– What do you think about people charged with murder? ____
– What do you think about this defendant? _______________
– How would feel if you were a juror in this trial? ___________
B. The format of the question – Closed-ended questions
– Close-ended (restricted) questions provide a limited number
of specific response alternatives. You can probably see how
the advantages and disadvantages differ from open-ended
questions …
Advantages: They can provide control over a wide range
of participant responses, reigning them in to just a few
Disadvantages: Information is not as rich as information
from open-ended questions
 3). Developing a good survey or questionnaire – Trial consulting
B. The format of the question – Closed-ended questions
– How often do you read the newspaper? (circle one):
a. Never b. Once a week c. Two to three times a week
d. Four to five times a week e. Six to seven times a week
– How many articles have you read about this case?
a. None b. One or two c. Three to five d. More than five
– Assess the following statement: “The defendant is accused
of killing his spouse while he was intoxicated” (circle one)
True False
B. The format of the question – Partial open-ended questions
– Partial open-ended questions, like close-ended questions,
give closed-ended options, but also give participants room
to add their own answer
– Make sure to add this response option, and that it …
Is clearly labeled
Has clear instructions
Provides a blank space, number, etc. consistent with
other items. Consider my “Other ________” option …
B. The format of the question – Partial open-ended questions
– “What piece of information would be most important in
finding this defendant not guilty? (check one)”
___ He is a good father
___ He doesn’t have a police record
___ He has received awards for his charitable work
___ He had a right to kill a cheating spouse
___ Other (Specify) ____________________________
B. The format of the question – Rating scales
– Rating scales are a lot like close-ended questions, but they
use scales rather than categorical responses
– Setting the number of ratings and labeling is flexible
No set minimum number, but 5 to 10 options is best
Labeling clarifies the scale
– Likert scales are actually a special type of scale that uses
disagree to agree ratings (bipolar in orientation). Yet many
scales use a similar format (Likert-type scales)
B. The format of the question – Rating scales
– How confident are you the defendant committed this crime?
1 2 3 4 5 6 7 8 9
Not at all confident Neutral Very confident
– How likely would you be to sentence the defendant to die?
1 2 3 4 5 6 7 8 9
Not at all Likely Very Likely
C. Assembling your questionnaire
– Organizing surveys takes some skill, and it is almost artful. It
should be straightforward, coherent, and visually pleasing
– The first question should “grab” the respondent
Should be interesting and engaging
Should apply to all respondents
– The questionnaire should have continuity, flowing from one
question to the next in a logical manner (all demographics
questions together, all general legal attitudes together, etc.)
 Answer the following two questions using the scale below
1. How happy are you with your life in general?
2. How happy are you with your dating / marriage?
1 2 3 4 5 6 7 8 9 10 11
Not So Extremely
Happy Happy
 Now imagine saw the two options presented this way …
1. How happy are you with your dating / marriage?
2. How happy are you with your life in general?
1 2 3 4 5 6 7 8 9 10 11
Not So Extremely
Happy Happy
C. Assembling your questionnaire – Question Order
– As you saw, the order of the questions was reversed in the
prior slide compared to the one before it
The first slide had the “general life” questions first and
the “dating / marriage” question second
The second slide had the “dating / marriage” question
first and the “general life” question second
Do you think this ordering might impact respondents?

C. Assembling your questionnaire – Question Order
– Actual research by Schuman and Presser (1981) shows that
People asked the dating or marriage question FIRST
report that they are happier with their life in general
Apparently, focusing on something positive (like
dating or marriage) primed people to think more
highly of their life in general
– Prior questions inform decisions on later questions, and thus
researchers must be very aware of question order effects
C. Assembling your questionnaire – Type of Question Order
– To avoid biasing respondents with closed-ended questions,
you might want to ask open-ended questions first. You can
then give close-ended questions later
– Yet some surveys might want to begin with closed-ended
questions to get key information that is easily comparable
with other survey participants, and then probe for more
information with open-ended questions later.
– The choice is up to the researcher
C. Assembling your questionnaire – Survey Continuity
– You also want to have some continuity to the survey
Ask about participants’ general crime attitudes in open-
ended questions first, and then case specific attitudes
Repeat this with the closed-ended questions (focus on
general questions first and then move to more specific
attitudes second)
C. Assembling your questionnaire – Sensitive questions
– Be wary when it come to asking sensitive questions
If you need to ask people embarrassing questions, it is
best to hold these off until last if you really need the info
Once they answer prior questions, it is easier to get
them to answer sensitive questions later, as those
questions may not seem as objectionable
C. Assembling your questionnaire – Navigational guides
– You may also want to use navigational guides
Guides direct
people to answer
only questions
applicable to
them, like this
museum survey
C. Assembling your questionnaire – Navigational guides
– Did you read any articles about this defendant?
If no, please skip to question 12
If yes, did the information in the articles seem to support
the defense, the prosecution, neither, or both?
If neither, put an X here ___ and skip to question 12
If both, put an X here ____ and skip to question 17
If only the prosecution, put an X here _____ and …
If only the defense, put an X here ______ and …
D. Administering your questionnaire
– How would you administer your trial consulting survey, and
why would you use that method? Again, a matter of choice
A matter of cost (internet is cheap, in-person expensive)
A matter of how much information you actually want
(more info requires longer surveys and more time)
It may depend on the complexity of the questions
It may depend on the sample you want to get (real jurors
or merely a sample of the population)
E. Characteristics of tests and inventories – Validity
– Validity refers to whether a survey is measuring what it
intends to measure. There are several kinds of validity …
1. Content validity
2. Construct validity
3. Concurrent validity
4. Criterion validity
– 1. Content validity refers to whether the questions cover the
whole range of behaviors normally considered to be a part
of the dimension you are assessing
For example, a depression scale that only measures the
affective (emotional) dimension lacks content validity (it
does not measure the behavioral dimension, which
experts say is a crucial component of depression)
Experts may have to agree on this (inter-rater reliability)
– 2. Construct validity refers to whether operational definitions
accurately reflect the true theoretical meaning of the concept
If you are looking at human intelligence, you may define
intelligence in terms of language proficiency
You lack construct validity if measuring
intelligence by the amount of popcorn a
person can eat
– 3. Concurrent validity correlates your results with another
survey that looks at the same dimension
Consider the SAT and ACT. You took one (or both) prior
to applying to FIU. Both assess your academic ability,
predicting your chances of doing well in college.
If you correlated these two tests, the concurrent validity
question is, “Do they agree on your college potential?” If
they agree, currently validity is high! They “concur”.
– 4. Criterion validity is like concurrent validity, but it focuses
on a comparison of one score with a future score on a test
or inventory
Are you thinking of applying to graduate school? If so,
you are probably thinking about the GRE. The GRE is
supposed to predict your success in graduate school.
Does it? Would your GRE score correlate with your
future graduate school GPA? Is so, you have good
criterion validity!
– IMPORTANT NOTE:
If you want to learn more about validity characteristics
(criterion, concurrent, content, and construct validity),
read Salkind Chapter 6 (Just The Truth). It’s an optional
chapter, but there is a powerpoint presentation set up for
the chapter! It has a fun, sex-oriented example of you
read it (but you’ll also learn more about validity)!
E. Characteristics of tests and inventories – Reliability
– Reliability refers to the extent to which a test or inventory is
consistent in evaluating the same individuals (repeatable)
– Again, there are lots of different types of reliability, but some

are specific to surveys …
1. Test-retest reliability
2. Parallel-forms reliability
3. Split-half reliability
– 1. In test-retest reliability, you administer the questionnaire,
wait, re-administer it again, and then compare the scores
If scores are similar, it has high test-retest reliability
– 1. In test-retest reliability, you administer the questionnaire,
wait, re-administer it again, and then compare the scores
If scores are similar, it has high test-retest reliability
A problem here is that participants might recall their time

one survey, which influences their time two responses. It
also requires multiple questionnaire sessions
How might we counter these problems?
– 2. With parallel forms reliability, the researcher creates two
comparable versions of the survey, and then give the two
forms to either the same person twice or different people
Similar scores on similar but different questions shows
parallel forms reliability, but are questions really equal
Are the following two questions really equivalent?
• “Do you support the death penalty?”
• “Would you be willing to give the death penalty?”
– 3. Split-half reliability is a third option in which researchers
split the same survey into equivalent halves and administer
both halves at the same time to the same participant
They then compare participant answers on the first part
of the survey to answers on the second half, or compare
all of the odd numbered questions to the even numbered
questions, etc. You get the survey done in one session,
but it still takes a long time and might feel repetitive. Yet
you can establish split-half reliability easily!
– IMPORTANT NOTE:
I just wanted to reiterate that if you want to learn more
about reliability characteristics (test-retest, parallel forms,
etc.), read Salkind Chapter 6 (Just The Truth). It’s an
optional chapter, but there is a powerpoint presentation
set up for the chapter! It has a fun, sex-oriented example
of you read it (but you’ll also learn more about reliability)!
F. Types of tests and inventories
– Your textbook does a good job covering these …
1. Achievement tests evaluate individuals’ level of
mastery or competence (like the bar exam for lawyers)
2. Aptitude tests assess potential ability or skill for a job
(like the GRE, which many of you all are taking soon!)
3. Personality tests and inventories measure aspects of
a person’s motivational state, capability, or personality
(like the MMPI or Beck Depression Inventory)
G. Ethical issues in survey research include …
– Anonymity, or guaranteeing that there will be no way for
participant’s names to be associated with their answers
This can be accomplished by using coding schemes,
coding participants’ identities as separate numbers, or,
separating informed consent forms from answers
With anonymity, the researcher does not know which
participant answered the survey, protecting their identity
Contrast this with …
G. Ethical issues in survey research include …
– Confidentiality, where the researcher can identify individuals
but does not disclose that personal information
Any data they do report may be reported in an aggregate
format or names of the participant are altered
Part Four
Sampling Considerations
 In Part Four, we explore some of the characteristics of sampling in
non-experimental designs, including …
1). The basics of sampling and representativeness
2). Sampling techniques
3). Other sampling issues
 1). The basics of sampling and representativeness
Before getting too far into this section, let’s better define what
we mean by a population versus a sample
– A population is the complete set of individuals / events
– A sample is a group that is selected from the population to
represent the population
– In an ideal world, we would study the population. After all,

why look at a sample and infer population characteristics
when we can just look at the actual population! Well …
It is difficult to study a whole population (which is not only hard
to access, but expensive as well, which is why we sample)
– The US Census survey tries to study all individuals in the
US population every 10 years, but even this survey misses
people (it doesn’t count the WHOLE population, which it is
designed to count!)
Ironically, the Census Bureau once proposed sampling,
but this new way of counting the number of US citizens
was denied based on constitutional grounds (the US
constitutional requires a population count!)
When we do sample, we seek a representative sample that
closely matches the characteristics of the population
– A non-representative sample is a biased sample
There are several sampling techniques. We’ll get them soon
For now, recall our golf ball example. Imagine you have a bag
of 300 balls: 100 white, 100 yellow, and 100 red (population)
– If you drew 30 golf balls at random from the bag, how many
of each color should you have in your sample?
10 white, 10 yellow, 10 red would be representative
There are several sampling techniques. We’ll get them soon
For now, recall our golf ball example. Imagine you have a bag
of 300 balls: 100 white, 100 yellow, and 100 red (population)
– If you drew 30 golf balls at random from the bag, how many
of each color should you have in your sample?
If you drew 25 white, 1 yellow, and 4 red, it would not be
representative, and is actually quite biased!
But it is possible to get this biased sample, even through
random sampling.
So can we make sure we get a more representative sample?
– Absolutely! There are a few techniques researchers can use
to get a representative sample, including
A. Random Sampling
B. Stratified Sampling
C. Proportionate Sampling
D. Single-Strata Sampling
E. Cluster Sampling
 2). Sampling techniques
A. Random sampling is … random. Every population member
has an equal likelihood of being included in the study
– Think about the golf ball bag (population of 300). Drawing a
sample of 30 balls involves just dipping your hand into the
golf bag 30 times and grabbing a single ball, one at a time
Similar to random assignment, we hope our sample is
representative (but it is random, leaving this to chance)
A. Random sampling is … random. Every population member
has an equal likelihood of being included in the study
– Researchers can obtain a random sample through:
Random digit dialing (common for telephone surveys)
Mass mailings (either snail-mail or internet surveys)
– Like random assignment, the more participants, the more
representative a random sample is. Yet just as matched-
pairs designs guarantee equal study conditions, increasing
representativeness is possible through …
B. Stratified sampling provides a way to better guarantee that
all aspects of the population as represented in the sample
– For this procedure, the researcher divides the population
into segments, or strata
– She then selects a sample of equal size from each strata
Imagine trying to find a stratified sample among Earth’s
population in the year 3000 where the earth is full of
humans, robots, aliens, and mutants (this is where my
love for the dearly departed Futurama TV series really
comes in!) …
 2). Sampling techniques – A “Futurama” strata from the year 3000!
B. Stratified sampling provides a way to better guarantee that
all aspects of the population as represented in the sample
– Each of our four “races” are equally represented in our study
25% Human 25% Robot 25% Alien 25% Mutant

C. Proportionate sampling is very similar to stratified sampling
in that it also divides the population into segments, or strata
– With proportionate sampling, though, researchers select a
sample from each stratum that is proportionate to the
percentage of stratum in the actual population
– Think about “Futurama” again. Using stratified sampling, we

had 25% of our sample for each strata. But do you think our
Earth will have equal 25%, 25%, 25%, and 25%? Probably
not, so let’s get more specific with proportionate sampling …
 2). Sampling techniques – A “Futurama” proportionate sample!
C. Proportionate sampling is very similar to stratified sampling
in that it also divides the population into segments, or strata
– 1000 years from now, chances are high there will be more
humans than aliens and mutants (though maybe not robots!)
If Earth has 75% humans, then a proportionate sample
should have 75% humans
 2). Sampling techniques – A “Futurama” single strata example
D. In single strata sampling, the researcher gathers gather data
from a single stratum of the population of interest
– Maybe you have no interest in robots, aliens, or mutants,
and the only humans you are interested in are humans who
are over 150 years old
– You could sampling ONLY
this strata for your study
E. Cluster sampling focuses on specific groups of participants
of interest to the researcher. Consider the following groups …
– Group A (Humans)
– Group B (Robots)
– Group C (Aliens)
– Group D (Mutants)
If you are only interested in organic organisms that
began life on Earth, you can cluster sampling from
Groups A and D only.
 3). Other sampling issues
Regardless of whether the researcher uses random sampling,
stratified sampling, proportionate sampling, single strata, or
cluster sampling, they have to decide if they can (or should)
reuse participants who they have already drawn
– In sampling without replacement, the researcher chooses a
participant (a golf ball, a score, an event) and then does not
allow that participants to be to be selected again
– In sampling with replacement, the participant is returned to
the population (the golf bag or the original population) and
can be selected again
Regardless of whether the researcher uses random sampling,
stratified sampling, proportionate sampling, single strata, or
cluster sampling, they have to decide if they can (or should)
reuse participants who they have already drawn
– There are statistical implications involved in dealing with the
with versus without replacement techniques, which go
beyond this course, but if you are curious there are some
good websites you can visit for more information, including:
– https://www.thoughtco.com/sampling-with-or-without-
replacement-3126563
Finally, a different kind of sampling technique (and one that is
fairly common) is non-probability sample, which does not focus
on representativeness but rather on convenience
– In non-probability sampling, you survey the first 5, 10, 25,
50, or 1000 people that you see
News organizations often do this to get survey data fast
Contrast this with probability (random) sampling, where
every person in the population has an equal chance of
being chosen
Part Five
Basic Research Strategies

 In our final section in this chapter, we will discuss some basic
research strategies, including …
1). The cross-sectional design
2). The longitudinal design
3). The cohort-sequential design
For those of you interested in developmental psychology or

gerontology, really pay attention to these designs!
 1). The cross-sectional design
In cross-sectional designs, researchers focus on participants
from different age groups at the same time
– The researcher creates “cohort” groups based on different
participant age groups
A researcher might study intelligence in five-year-olds,
ten-year-olds, and fifteen-year olds at the same time to
glimpse the development of logical thought processing
– The neat thing about this design is it allows developmental
data collection in a short time period, since you look at more
than one age group simultaneously
The Cross-Sectional Design
Here, we have 5 year olds,

10 year olds, and 15 year olds
participating in the same
study at the same time
 1). The cross-sectional design – Concerns
There are some problems with cross-sectional designs
– These designs may not be appropriate using wide ranging
age groups, as the groups may differ too much
Comparing five year olds, ten year olds, and fifteen year
olds might be okay, but comparing 10 year olds, thirty
year olds, and 50 year olds might be a harder sell, as
these varying ages differ a lot
Life experiences for older participants may impact
their intelligence, making it hard to compare them to
much younger participants
 1). The cross-sectional design – Concerns
There are some problems with cross-sectional designs
– These designs may not be appropriate using wide ranging
age groups, as the groups may differ too much
Relatedly, cross-generation effects are a problem
Educational opportunities for a 1990’s cohort differ
wildly from the opportunities for a 1970’s cohort, so
the intelligence levels of each cohort may disallow
easy comparisons
 2). The longitudinal design
If you want to see how a behavior really develops, you might
follow a single group of participants at several points in time
(weeks, months, or years) using a longitudinal design
– You might look at language in a three-year-old, then study
her again when she is four, then again when she is five
Can gain valuable insight into the behavior development
The Longitudinal Design: 1945-1995
Here, we you start looking at

5 year olds in 1945, then look
at the same kids in 1955,
1965, 1975, 1985, 1995, etc.
Of course, there are two big

issues with longitudinal
studies …
 2). The longitudinal design – Concerns
A. First, if you study the same person longitudinally, you might
develop good theories about him, but that data may not apply
to other generations
– After years following Mr. Burns, you know a lot about him,
but that won’t help you understand Bart!
B. Second, subject mortality may occur if you follow someone
over a long time period (either the participant actually dies OR
they do not come back for subsequent examinations)
– This is problematic, as the sample you have at the end of
the study may differ a lot from the one you started with.
C. Third, multiple observations may influence responses
– Changes you see in participant responses may be based on
their increasing experience in the study rather than changes
related to age
Using different stimuli versions or different kinds
of tests (e.g. parallel forms) to measure behavior
might help
But longitudinal designs have benefits, too …
 2). The longitudinal design – Benefits
First, these designs permit you to see developmental changes
Second, you can also witness the development of a behavior,
noting when, how, and why it comes into being
 3). The cohort-sequential design
Cohort-sequential designs combine both cross-sectional and
longitudinal component into the same research study
– The cohort-sequential design focuses on a cohort group like
a cross-sectional design (participants born at a specified
time) but follows them longitudinally and compares them to
other cohorts
For example, compare 5 year olds from 1980 to 5-year-
olds from 1990 to five-year-olds from 2000
– It allows you to test for, but not eliminate, generation effects
The Cohort-Sequential Design
Not only do you get to see the

development of a behavior, you
can compare it to other cohorts
Of course, it takes a long time to

complete (the longitudinal aspect)
and you still encounter mortality
and multiple observation issues,
but you get lots of good data!
Part Six
An Eye Toward The Future

 An Eye Toward The Future
Up next, I want to delve a bit further into scale development.
We’ll cover some of the same information in our next lecture
(Chapter 6, Salkind), but we’ll put more of a real-world twist on
it, looking at sexism!
 Finally, it is VERY, VERY, VERY important for you to read your lab
presentation immediately. Since many of your papers are based
on content covered in the lab, you need to know about that content
sooner rather than later
 So, here is your reminder to read that lab presentation immediately

Chapter 4 (Smith and Davis) : Non-Experimental Methods

Hochgeladen von

Dokumentinformationen

Originaltitel

Copyright

Verfügbare Formate

Dieses Dokument teilen

Dokument teilen oder einbetten

Freigabeoptionen

Stufen Sie dieses Dokument als nützlich ein?

Sind diese Inhalte unangemessen?

Copyright:

Verfügbare Formate

Chapter 4 (Smith and Davis) : Non-Experimental Methods

Hochgeladen von

Copyright:

Verfügbare Formate

Chapter 4 (Smith and Davis)

In this section, you will learn about …

Historical Accounts Military Data

Yet gaining access to archival data can be tricky

Of course, the source of the information in archival studies

– “Yankees Edge out Red Sox 8-7”

– “Yankees Bludgeon Red Sox 8-7”

– “Yankees Edge out Red Sox 8-7”

Second, the original author may have been selective in what

The goals of naturalistic observations are to

– C). Being objective can be a hurdle as well.

– Third (and related), we can ask experts about the best

But how do you find inter-rater reliability? Consider some data:

Percent agreement is an estimate: It can easily go up or down

– Recall the % agreement among our two psychologists …

– We will discuss the calculations for these a bit more in our

– But your sample may not be representative

Building survey questions …

– If I asked you the question, “How successful are you?”, what

– Do you think men would have responded that they would

– How likely would you be to sentence the defendant to die?

1. How happy are you with your life in general?

2. How happy are you with your dating / marriage?

1. How happy are you with your dating / marriage?

2. How happy are you with your life in general?

Do you think this ordering might impact respondents?

– Again, there are lots of different types of reliability, but some

A problem here is that participants might recall their time

– In an ideal world, we would study the population. After all,

25% Human 25% Robot 25% Alien 25% Mutant

– Think about “Futurama” again. Using stratified sampling, we

Basic Research Strategies

For those of you interested in developmental psychology or

Here, we have 5 year olds,

Here, we you start looking at

Of course, there are two big

Not only do you get to see the

Of course, it takes a long time to

An Eye Toward The Future

 So, here is your reminder to read that lab presentation immediately

Das könnte Ihnen auch gefallen