10/2/2011

Krishna Bista 10/04/2011

WHAT IS INTERNAL VALIDITY? Internal validity is a crucial measure in quantitative studies, where it ensures that a researchers experiment design closely follows the principle of cause and effect (Shuttleworth, 2009) Could there be an alternative cause, or causes, that explain my observations and results?
Low-income groups
Smoking

Internal validity: Are the methods correct and the results accurate?
Do the research conditions warrant the conclusions? Without internal validity results are un-interpretable. External validity: Are the findings generalizable beyond that particular study?
To what extent can the results be generalized? To what populations, settings, treatment variables, and measurement variables?
(Lunenburg & Irby, 2008)

10/2/2011

Do tests really measure what student learning? Do college GPAs accurately predict on the job success? Content validity: CSEQ should measure college experience, not depression! Face validity: African Americans take a history test, but all of the questions were written by Asians. Criterion validity: Do a persons reported income predict his or her credit score? Construct validity: E.g. scores on a marital satisfaction scale should be negatively related to spouse abuse. Predictive validity: Does the Suicide Probability Scale accurately predict which adolescents are likely to attempt suicide?
4

Cause and Effect Single-Group Design (variables) Multiple-Group Design (variables)

Temporal precedence

Cause

then

Effect

Time

Covariation of cause and effect

if X, then Y if not X, then not Y

No alternative explanations Alternative cause

Alternative cause

Program

Causes?

Outcome

Adopted from Brown (2010)

Alternative Alternative cause cause

6

10/2/2011

Alternative explanations

Measure baseline

Administer program

Measure outcomes

X
O =Observation/Measurement X= Treatment Intervention

O
Alternative explanations

Adopted from Brown

(2010)
7

Alternative explanations Measure baseline Administer program Measure outcomes

O O
Measure baseline

X
Do not administer program

O O
Measure outcomes Alternative explanations

Adopted from Brown (2010)

History Maturation Testing Instrumentation Statistical regression Selection Research mortality Interactions w/ selection
(Brown, 2010; Krathwohl & Smith, 2005; Rudestam, & Newton, 2001)

10/2/2011

Specific events (9/11,Tsumami, bomb, flooded schools, flu), during the life of the experiment The longer the interval between the pretest and posttest, the more viable this threat. Example: A Longitudinal Study of the Adaptation of
International Students in the United States. (Toni Falbo,
University of Texas at Austin, 2005)

10

Changes in physical, intellectual, or emotional characteristics of the participants In longitudinal studies (pre-test/pos-ttest/multi-test), for instance, individuals grow older, become more sophisticated, more/less motivated, become bored/anxious Example: Asian Students Voices: An Empirical Study of Asian Students' Learning Experiences at a New Zealand University (Campbell & Li, 2007).
Changes in language ability, communication skills, and cross-cultural adaptations

11

Also called pretest sensitization, this refers to the effects of taking a test upon performance on a second testing. Testing becomes a more viable threat to internal validity as the time between pretest and posttest is shortened. Example: Pre/Post/multi-6th grader -10th grader Does the test measure factual information that can be easily recalled or sth else?

12

10/2/2011

Changes in the way a test or other measuring instrument is calibrated that could account for results of a research study (e.g. Missouri Test). This threat typically arises from unreliability in the measuring instrument. Observer driftbeing bored Poor survey construction

13

Occurs when individuals are selected for an intervention or treatment on the basis of extreme scores on a pretest. Extreme scores are more likely to reflect larger (positive or negative) errors in measurement. Results move towards the mean

14

This can occur when intact groups are compared. The groups may have been different to begin with. E.g. Does watching American Idol increase singing in the shower?

15

10/2/2011

Occurs when differential selection is confounded with maturational effects. Example: dating behavior--girls who attended a dance school and who did not attend
Dance/Nondance Group Pubertal Status interaction was found. Breast Development Dance/Nondance Group interaction was found.
(Gargiulo, Attie, Brooks-Gunn, &Warren, 1987).

16

The differential loss of individuals from treatment and/or comparison groups. This is often a problem when research participants are volunteers. Volunteers may drop out if they find it is time consuming. Example: A Longitudinal Study of the Adaptation of
International Students in the United States. (Toni Falbo, University of Texas at Austin, 2005)

17

Whenever measuring affects DV score Behavior and self-report (faking)/Judgeaphobia May divert attention from experiment instructions to others As an example, three different classroomsstudents with how, low, and average skills (CSEQESL vs. Graduate
Students)

18

10/2/2011

Hide your identity (dont call yourself a psychologist/researcher) Be informal, friendly, put them at ease Control with self-report measures (peer-evaluation, anonymity, confidentiality/bogus pipeline)
(Rudestam & Newton, 2001)

19

Campbell, J., & Li, M. (2007). Asian students voices: An empirical study of Asian students' learning experiences at a New Zealand university. Journal of Studies in International Education, 12 (4), 375-396. doi:10.1177/1028315307299422 Brown, T. T. (2010). Research Design. HRH Labor Markets Course: Analyze and Plan Human Resources for Health Global Health Workforce Economic Network, University of California, Berkeley. Gargiulo, J., Attie, I., Brooks-Gunn, J., &Warren, M. (1987). Girls' dating behavior as a function of social context and maturation. Developmental Psychology, 23 (5), 730-737 Lunenburg, F. C., & Irby, B. J. (2008). Writing a successful thesis or dissertation: Tips and strategies for students in the social and behavioral sciences. Thousand Oaks, California: Sage Publications, Inc. Rudestam, K. E., & Newton, R. R. (2001). Surviving your dissertation: A comprehensive guide to content and process (2nd Ed.). Thousand Oaks, California: Sage Publications, Inc. Shuttleworth, M. (2009). Internal Validity. Retrieved from Experiment Resources: http://www.experiment-resources.com/internal-validity.html

20