Factors that threaten the validity of research findings Two classes of factors that jeopardize the validity of research

findings Factors concerned with internal validity. Do the research conditions warrant the conclusions? Without internal validity results are uninterpretable.

Factors concerned with external validity. To what extent can the results be generalized? To what populations, settings, treatment variables, and measurement variables?

Factors affecting Internal Validity Internal validity is threatened whenever there exists the possibility of un-controlled extraneous variables that might otherwise account for the results of a study. Eight classes of extraneous variables can be identified. History Specific events, in addition to the treatment, that occur between the first and second measurement. The longer the interval between the pretest and posttest, the more viable this threat. Maturation Changes in physical, intellectual, or emotional characteristics, that occur naturally over time, that influence the results of a research study. History Maturation Testing Instrumentation Statistical regression Selection Research mortality Interactions w/ selection

In longitudinal studies, for instance, individuals grow older, become more sophisticated, maybe more set in there ways. Testing Also called pretest sensitization, this refers to the effects of taking a test upon performance on a second testing. Merely having been exposed to the pretest may influence performance on a posttest. Testing becomes a more viable threat to internal validity as the time between pretest and posttest is shortened Instrumentation Changes in the way a test or other measuring instrument is calibrated that could account for results of a research study (different forms of a test can have different levels of difficulty). This threat typically arises from unreliability in the measuring instrument. Can also be present when using observers Statistical Regression Occurs when individuals are selected for an intervention or treatment on the basis of extreme scores on a pretest. Extreme scores are more likely to reflect larger (positive or negative) errors in measurement (chance factors). Such extreme measurement errors are NOT likely to occur on a second testing Differential Selection This can occur when intact groups are compared. The groups may have been different to begin with. If three different classrooms are each exposed to a different intervention, the classroom performances may differ only because the groups were different to begin with. Selection-Maturation Interaction Occurs when differential selection is confounded with maturational effects. The treatment group might be composed of higher aptitude students, or The treatment group might have more students who are born during the summer months.

Research Mortality The differential loss of individuals from treatment and/or comparison groups. This is often a problem when research participants are volunteers. Volunteers may drop our of the study if they find it is consuming too much of their time. Others may drop out if they find the task to be too arduous. INTERACTION OF SELECTION WITH THE OTHER FACTORS AFFECTING INTERNAL VALIDITY Occurs when intact groups, which may not be equivalent, are selected to participate in research interventions. As in a previous example, three different classrooms may be exposed to different treatments, but one of the classroom might be composed of students having higher achievement trajectories. External Validity Concerned with whether the results of a study can be generalized beyond the study itself: 1. Population validity (when the sample does not adequately represent the population). 2. Personological validity (when personal/ psychological characteristics interact with the treatment). 3. Ecological validity (when the situational characteristics of the study are not representative of the population). Factors affecting External Validity External validity is threatened whenever conditions inherent in the research design are such that the generalizability of the results is limited. Four classes of threats to external validity can be identified. Reactive or interactive effects of testing Interaction effect of selection bias and the intervention. Reactive effects of treatment arrangements Multiple treatment interference

Reactive effect of testing Occurs whenever a pretest increases or decreases the respondents sensitivity to the treatment.

Studies involving self-report measures of attitude and interest are very susceptible to this threat. Selection x Treatment Interaction This can occur when selected treatment or comparison groups are more or less sensitive to the treatment prior to initiating the treatment (or intervention). Most likely to occur when the treatment and comparison groups are not randomly selected. Reactive Effects of Experimental Arrangements These can occur when the conditions of the study are such that the results are not likely to be replicated in non-experimental situations. Hawthorn effects John Henry effects Placebo effects Novelty effect

Multiple-treatment Interference This has a likelihood of occurring whenever the same research participants are exposed to multiple treatments. Sequence effects Carry-over effects