RELIABILITY AND VALIDITY

LOUIS COHEN, LAWRENCE MANION & KEITH MORRISON

STRUCTURE OF THE CHAPTER

Defining validity Validity in quantitative research Validity in qualitative research Types of validity Triangulation Ensuring validity Reliability Reliability in quantitative research Reliability in qualitative research

STRUCTURE OF THE CHAPTER

Validity and reliability in interviews Validity and reliability in experiments Validity and reliability in questionnaires Validity and reliability in observations Validity and reliability in tests Validity and reliability in life histories

VALIDITY IN QUANTITATIVE AND QUALITATIVE RESEARCH

Validity in quantitative research often concerns: objectivity, generalizability, replicability, predictability, controllability, homothetic statements. Validity in qualitative research often concerns: honesty, richness, authenticity, depth, scope, subjectivity, strength of feeling, catching uniqueness, idiographic statements.

TYPES OF VALIDITY
Concurrent Consequential Construct Content Criterion-related Convergent & discriminant Cross-cultural Cultural validity Descriptive Catalytic Ecological Evaluative External Face Internal Interpretive Jury Predictive Systemic Theoretical

VALIDITY IN QUANTITATIVE RESEARCH

Concurrent Construct Content Criterion-related Convergent & discriminant Cross-cultural Evaluative External Face Internal Jury Predictive Theoretical

HISTORY DIRECTION OF CAUSALITY

MATURATION TESTING

TYPE 1 AND TYPE 2 ERRORS

THREATS TO VALIDITY AND RELIABILITY

INSTRUMENTATION

EXPERIMENTAL MORTALITY

OPERATIONALIZATION
CONTAMINATION REACTIVITY

ESTABLISHING VALIDITY IN QUALITATIVE RESEARCH

Prolonged engagement in the field Persistent observation Triangulation Leaving an audit trail Respondent validation Weighting the evidence (giving priority) Checking for representativeness Checking for researcher effects Making contrast/comparisons Theoretical sampling Checking the meaning of outliers Using extreme cases

ESTABLISHING VALIDITY IN QUALITATIVE RESEARCH

Ruling out spurious relations Replicating a finding Referential adequacy Following up surprises Structural relationships Peer debriefing Rich and thick description Looking for possible sources of invalidity Assessing rival explanations Negative case analysis Confirmatory data analysis Effect sizes

THREATS TO VALIDITY IN QUANTITATIVE RESEARCH

History Maturation Statistical regression Testing Instrumentation Selection Bias Experimental mortality Instrument reactivity Selection-maturation interaction Type I and Type II errors

THREATS TO EXTERNAL VALIDITY IN QUANTITATIVE RESEARCH

Failure to describe independent variables explicitly Lack of representativeness of available and target populations Hawthorne effect Inadequate operationalizing of dependent variables Sensitization/reactivity to experimental/research conditions Interaction effects of extraneous factors and experimental/ research treatments Invalidity or unreliability of instruments Ecological validity Multiple treatment validity

THE HAWTHORNE EFFECT

Between 1927 and 1932 researchers carried out experiments at the Western Electric Companys Hawthorne plant. Purposes: To examine the effects of changes of working conditions on output of workers Sample: Six women, chosen as average workers Method: Women worked in a test room. Output measured under different conditions (e.g. no change change to method of payment introduce two rest periods introduce six rest periods changes in lighting conditions, early clocking-off, five-day working week return to initial conditions Duration: 15 weeks

THE HAWTHORNE EFFECT

Results: Output rose steadily during test period and after the test period. Conclusion: Output did not seem to depend on test conditions. Increased output seemed to be due to the fact that the people had been involved in the experiment itself, i.e. the act of research had affected the results. The results were a research of the research itself. Implications: The act of being involved in research itself affects the results.

THREATS TO EXTERNAL VALIDITY IN QUALITATIVE RESEARCH

Selection effects Setting effects History effects Construct effects

ENSURING VALIDITY AT THE DESIGN STAGE

Choose an appropriate time scale; Ensure adequate resources for the research Select appropriate methodology Select appropriate instruments Use an appropriate sample Ensure reliability Select appropriate foci Avoid having biased researcher(s)

ENSURING VALIDITY AT THE DATA COLLECTION STAGE

Reduce the Hawthorne effect Minimize reactivity Avoid drop-out rates amongst respondents Take steps to avoid non-return of questionnaires Avoid too long or too short an interval between pre-tests and post-tests Ensure inter-rater reliability Match control and experimental groups Ensure standardized procedures for gathering data Build on the motivations of respondents Tailor instruments to situational factors Address researcher characteristics

ENSURING VALIDITY AT THE DATA ANALYSIS STAGE

Use respondent validation; Avoid subjective interpretation of data Reduce the halo effect Use appropriate statistical treatments Recognize extraneous factors which may affect data Avoid poor coding of qualitative data Avoid making inferences/generalizations beyond the data Avoid equating correlations and causes Avoid selective use of data Avoid unfair aggregation of data Avoid degrading the data; Avoid Type I and/or Type II errors

ENSURING VALIDITY AT THE DATA REPORTING STAGE

Avoid using data selectively and unrepresentatively Indicate the context and parameters of the research Present the data without misrepresenting the message Make claims which are sustainable by the data Avoid inaccurate or wrong reporting of data Ensure that the research questions are answered Release research results neither too soon nor too late

RELIABILITY IN QUANTITATIVE AND QUALITATIVE RESEARCH

Reliability in quantitative research: consistency (stability), accuracy, predictability, equivalence, replicability, concurrence, descriptive and causal potential. Reliability in qualitative research: accuracy, fairness, dependability, comprehensiveness, respondent validation, checkability, empathy, uniqueness, explanatory and descriptive potential, confirmability.

 Reliability as stability: Consistency over time and samples; Reliability as equivalence: Equivalent forms of same instrument; Inter-rater reliability; Reliability as internal consistency: Split half reliability (e.g. for test items)

TYPES OF RELIABILITY IN QUANTITATIVE RESEARCH

TRIANGULATION
Methodologies Instruments Researchers Time Location Theories Samples Participants Data

SPLIT-HALF RELIABILITY (Spearman-Brown)

Reliability =

2r 1 r

r = the actual correlation between the two halves of the instrument (e.g. 0.85); Reliability =

2 ( 0.85 ) 1 0.85

1.70 185

= 0.919

RELIABILITY IN QUALITATIVE RESEARCH

Credibility Neutrality Confirmability Dependability Consistency Applicability Trustworthiness Transferability

RELIABILITY AND REPLICATION IN QUALITATIVE RESEARCH

Repeat:
The status position of the researcher The choice of informants/respondents The social situations and conditions The analytic constructs used The methods of data collection and analysis

Address:
Stability of observations Parallel forms Inter-rater reliability Respondent validation

IMPROVING RELIABILITY
Minimise external sources of variation; Standardise conditions under which measurement occurs; Improve researcher consistency; Broaden the sample of measurement questions by: a) adding similar questions to the instrument; b) increasing the number of researchers (triangulation); c) increasing the number of occasions in an observational study. Exclude extreme responses (outliers).

RELIABILITY AND VALIDITY AT ALL STAGES

Design and methodology Sampling Instrumentation Timing Data collection Data analysis Data reporting