Chapter 11

Experimental Designs

Power Point Slides by Ronald J. Shope in collaboration with John W. Creswell

Key Ideas
Definition and uses for experimental designs Procedures used in experimental research Internal and external validity threats in experimental research Types of experimental designs Steps in conducting experimental research Evaluating experimental research

Educational Research 2e: Creswell

What is experimental research?

In an experiment, you test an idea (or practice or procedure) to determine whether it influences an outcome or dependent variable. Determine activities with which to experiment Assign individuals to experience it (and have some individuals experience something different) Determine whether those who experienced the activities (or practice or procedure) performed better on some outcome than those who did not experience it.

Educational Research 2e: Creswell

When do you use experimental procedures?

To establish cause and effect
Between independent and dependent variables Control for all variables that might influence the outcome

Used when testing theories Used when comparing two or more groups
Educational Research 2e: Creswell

How did experiments develop?

Psychological experiments (late 19th early 20th century) Procedures for comparing groups (McCall 1925) Statistics for comparing groups (e.g. Chi-Square) Identification of types of experimental designs (Campbell and Stanley 1963) Types of basic designs and threats (Cook and Campbell 1979) Complicated experiments with many variables (Since 1980)
Educational Research 2e: Creswell

Key Characteristics of Experimental Designs

Random assignment Control over extraneous variables Manipulation of the treatment conditions Outcome Measures Group Comparisons Threats to validity

Educational Research 2e: Creswell

Random assignment

Randomly assign individuals at random

conditions or to groups Random assignment equates groups and distributes variability between or among groups and conditions Different from Random Selection which is selection of participants at random

Educational Research 2e: Creswell

Control over extraneous variables

Extraneous Variables: influences in participant selection, procedures, statistics, or the design likely to affect the outcome and provide an alternative explanation results than what was expected. Random assignment helps to control for extraneous variables Done before the experiment begins

Educational Research 2e: Creswell

Control over extraneous variables

Other control procedures

pretest/posttest covariates matching participants selecting homogenous samples using blocking variables

Educational Research 2e: Creswell

Pre-Test and Post-Tests

Intervention
Time 1
Pre-Test

Time 2
Post-Test

Educational Research 2e: Creswell

Controlling for Covariates

No Covariates
Independent Variable Dependent Variable

Covariate Introduced
Independent Variable: Type of Instruction Dependent Variable: Rates of Smoking Covariate: Parents Who Smoke

Variance
Educational Research 2e: Creswell

Variance Removed

Matching Process Based on Gender

John Jim James Josh Jackson Jane Johanna Julie Jean Jeb
Educational Research 2e: Creswell

Experimental Group

Control Group

Manipulation of the treatment conditions

Identify a treatment variable Identify the conditions or levels of the treatment variable Manipulate the treatment conditions

Educational Research 2e: Creswell

The Experimental Manipulation of a Treatment Group

Independent Variables
1. Age (cant manipulate) 2. Gender (cant manipulate) 3. Types of Instruction (can manipulate) a. Lecture (control) b. Lecture + Hazard Instruction (Comparison) c. Lecture + Hazard Instruction + slides of damaged lungs (experiment)
Educational Research 2e: Creswell

Dependent Variable Frequency of Smoking

Outcomes
Criterion or effect variable Outcome variable Measured on a continuous scale

Educational Research 2e: Creswell

Group comparisons in an experiment

Phase 1: Relationship Picture + Error Correction Treatment Phase 2: Timeline Picture
Class A: Regular Spelling Practice (Control) Class B: Reduced word list (Comparison) Class C: Error Correction (Experimental)

Spelling Accuracy
Test 1 Test 2 6 Weeks 6 Weeks 6 Weeks 6 Weeks Test 3 6 Weeks 6 Weeks

Error Correction Treatment

6 Weeks 6 Weeks Class C F-value 9.9 0.27 (3.9)
13.9 (4.2) 13.1 (3.8) 4.90* 3.31*

Phase 3: Statistical Comparisons

Test 1 Class A 10.3 (3.6) Class B 10.8 (4.3)

6 Weeks

Test 2
Test 3

10.7 (3.3) 11.1 (3.3)

10.6 (3.8)
10.3 (3.6)

* p<.05

Educational Research 2e: Creswell

Threats to internal validity

History Maturation Regression Selection Mortality Interactions with selection Diffusion of treatments Compensatory equalization Compensation rivalry Resentful demoralization Testing Instrumentation

Educational Research 2e: Creswell

Threats to external validity

Interaction of selection and treatment Interaction of setting and treatment Interaction of history and treatment

Educational Research 2e: Creswell

Types of experiments: Between groups

True Experiments
Pre- and posttest Posttest Only

Quasi Experiments
Pre Posttest Factorial Designs

Educational Research 2e: Creswell

Means and Main Effects of Eight groups in Factorial Design

Depression
Low
Health lecture

Medium High Main Effects of Type of Instruction

Type of Instruction
Standard lecture

Mean rate Mean rate Mean rate of of of smoking smoking smoking Mean rate Mean rate Mean rate of of of smoking smoking smoking

Main Effects of Depression

Educational Research 2e: Creswell

Graphs Showing Main and Interaction Effects

Standard lecture (a) No Interaction Effects (Parallel) High Health lecture

Extent of Smoking
Low High Low Medium High Standard lecture Health lecture High Standard lecture Health lecture Low Low Medium High

(b) Interaction Effects (Crossed)

Extent of Smoking
Low Low Medium

(c) Interaction Effects (Not Parallel)

High

Extent of Smoking

Educational Research 2e: Creswell

Types of Experiments: Withingroup or individual Time series experiments

interrupted uninterrupted

Repeated measures experiments Single-subject experiments

A/B design Multiple baseline design Alternating treatments
Educational Research 2e: Creswell

Steps in Conducting Experimental Research

Decide if an experimental design fits research problem Form cause/effect hypotheses Select experimental unit and identify study participants Select an experimental treatment and introduce it Choose a type of experimental design Conduct the experiment Organize and analyze the data Develop an experimental research report
Educational Research 2e: Creswell

Criteria for Evaluating Experimental Research

Does the experiment have a powerful intervention? Does it employ few treatment groups (e.g. only two)? Will participant gain from the intervention(s)? Is there a systematic way the researcher derived the number of participants (cell size)? Was there an adequate number of participants used in the study? Were valid, reliable, and sensitive measures or observations used? Did the study control for extraneous factors? Did the researcher control for threats to internal validity?

Educational Research 2e: Creswell