Validity and Reliability

Bihungum Bista
Research Officer
Nepal Health Research Council
 Validity

• Validity is the extent to which a test measures

what it is supposed to measure.

1/31/2020 2
 Validity
• Internal
– Is the experimenter measuring the effect of the
independent variable on the dependent
variable?

• External
– Can the results be generalised to the wider
population?

1/31/2020 3
• Errors are generally produced by one
or more of the following:

a) Random error
b) Systematic error(Bias)
-selection bias
-Information bias
-Confounding
 Control measures
• At design:
- Randomization
- Matching
- Confounding
• At analysis phase
- Stratification
- Modeling
1/31/2020 5
Types of validity
• Face validity
Measurement, or question, makes sense to you,
interviewers, experts, subjects, et al.
• Content validity is a logical process where
connections between the test items are checked.
• Criterion validity
Measurement agrees with a “gold standard”.
• Construct validity
Measurement agrees with other operational
measurements of the same concept.
Example: depression
1/31/2020 6
 Face Validity
Do the questions look like they measure what
they are supposed to?
• What does the question below look like it
would measure? If you said depression you
would be correct. It looks like
an item from a depression
Choose the item that best describes you: scale – and it is!
(0) I do not feel sad.
It has face validity.
(1) I feel sad.
(2) I am sad all the time and I can't snap out of it. If someone said this was
(3) I am so sad or unhappy that I can't stand it. from a parental attachment
scale, then you could say that
it lacks face validity.
1/31/2020 7
 Content validity
• Content validity is a logical process where
connections between the test items are
checked.
• Does the test contain items from the desired
“content domain”?
• Content validity is not “tested for”. Rather
it is “assured” by the informed item
selections made by experts in the domain.
1/31/2020 8
 Predictive Validity
Does the measure predict something that it
logically should?
The SAT
According to their website, “the SAT is the one test that shows
colleges what you know”. College admissions officers use this test
to determine who will do well in college.
What future measures do you expect that the SAT would be
correlated with?
If you said grades in college you would be right, and the SAT IS
correlated with that! – somewhat…
It has predictive validity
1/31/2020 9
 Concurrent Validity
Does this measure correlate strongly with
something that it logically should right now?
Unlike predictive validity, concurrent validity compares measures
taken at the same time.
What current measures do you imagine that GPA
should correlate with?
If you said current test scores and teacher ratings
you would be correct.
GPA has some concurrent validity
1/31/2020 10
 Convergent and Discriminant
Validity

1/31/2020 11
 The Convergent Principle
Measures of constructs that are related
to each other should be strongly
correlated.

1/31/2020 12
 How It Works
Self-esteem
Theory construct

Item 1 Item 2 Item 3 Item 4

1/31/2020 13
 Theory
How It Works
You theorize that the Self-esteem
construct
items all reflect self-esteem.

Item 1 Item 2 Item 3 Item 4

1/31/2020 14
 How It Works
Theory Self-esteem
construct

Item 1 Item 2 Item 3 Item 4

1.00 .83 .89 .91

.83 1.00 .85 .90
.89 .85 1.00 .86
.91 .90 .86 1.00

Observation
1/31/2020 15
 How It Works
Theory Self-esteem
construct

Item 1 Item 2 Item 3 Item 4

1.00 .83 .89 .91

.83 1.00 .85 .90
.89 .85 1.00 .86
.91 .90 .86 1.00

The correlations provide evidence

Observation that the items all converge
1/31/2020 16
on the same construct.
 The Discriminant Principle
Measures of different
constructs should not
correlate highly with each
other.

1/31/2020 17
 How It Works
Theory
Self-esteem Locus-of-control
construct construct

SE1 SE2 LOC1 LOC2

1/31/2020 18
 You theorize that you have two
distinguishable constructs.
Theory
How It Works
Self- esteem Locus-of-control
construct construct

SE1 SE2 LOC1 LOC2

1/31/2020 19
 How It Works
Theory
Self-esteem Locus-of-control
construct construct

SE1 SE2 LOC1 LOC2

rSE1, LOC1 = .12

rSE1, LOC1 = .09
rSE1, LOC1 = .04
Observation
1/31/2020 rSE1, LOC1 = .11 20
 Theory
How It Works
Self-esteem Locus-of-control
construct construct

SE1 SE2 LOC1 LOC2

The correlations provide rSE1, LOC1 = .12

evidence that the items on
the two tests discriminate. rSE1, LOC1 = .09
rSE1, LOC1 = .04
Observation
1/31/2020 rSE1, LOC1 = .11 21
 Reliability
Reliability is the degree to which a test consistently
measures whatever it measures

1/31/2020 22
 Reliability or Consistency of
What?
• Observers or raters
• Tests over time
• Different versions of the same test
• A test at one point in time

1/31/2020 23
 Inter-Rater or Inter-Observer
Reliability

Object or
phenomenon

1/31/2020 24
 Inter-Rater or Inter-Observer
Reliability

Object or
phenomenon

Observer 1

1/31/2020 25
 Inter-Rater or Inter-Observer
Reliability

Object or
phenomenon

Observer 1 Observer 2

1/31/2020 26
 Inter-Rater or Inter-Observer
Reliability

Object or
phenomenon

?
=
Observer 1 Observer 2

1/31/2020 27
 Inter-Rater or Inter-Observer
Reliability
• Are different observers consistent?
• Can establish this outside of your study in a
pilot study.
• Can look at percent of agreement
(especially with category ratings).
• Can use correlation (with continuous
ratings).

1/31/2020 28
 Test-Retest Reliability

Time 1 Time 2

1/31/2020 29
 Test-Retest Reliability

Test
= Test

Time 1 Time 2

1/31/2020 30
 Test-Retest Reliability

Stability over time

Test
= Test

Time 1 Time 2

1/31/2020 31
 Test-Retest Reliability
• Measure instrument at two times for
multiple persons.
• Compute correlation between the two
measures.
• Assumes there is no change in the
underlying trait between time 1 and time 2.

1/31/2020 32
 Parallel-Forms Reliability

Time 1 Time 2

1/31/2020 33
 Parallel-Forms Reliability

Form A

=
Form B

Time 1 Time 2

1/31/2020 34
 Parallel-Forms Reliability

Form A Stability across forms

=
Form B

Time 1 Time 2

1/31/2020 35
 Parallel-Forms Reliability
• Administer both forms to the same people.
• Get correlation between the two forms.
• Usually done in educational contexts where
you need alternative forms because of the
frequency of retesting and where you can
sample from lots of equivalent questions.

1/31/2020 36
 Internal Consistency Reliability
Average inter-item correlation

1/31/2020 37
 Internal Consistency Reliability
Average Inter-Item Correlation

Test

1/31/2020 38
 Internal Consistency Reliability
Item 1 Average Inter-Item correlation

Item 2

Item 3
Test
Item 4

Item 5

Item 6

1/31/2020 39
 Internal Consistency Reliability
Item 1 Average inter-item correlation

Item 2 I1 I2 I3 I4 I5 I6

I1 1.00
Item 3
I2 .89 1.00
Test I3 .91 .92 1.00
Item 4 I4 .88 .93 .95 1.00
I5 .84 .86 .92 .85 1.00
I6 .88 .91 .95 .87 .85 1.00
Item 5

Item 6

1/31/2020 40
 Internal Consistency Reliability
Item 1 Average inter-item correlation

Item 2 I1 I2 I3 I4 I5 I6

I1 1.00
Item 3
I2 .89 1.00
Test I3 .91 .92 1.00
Item 4 I4 .88 .93 .95 1.00
I5 .84 .86 .92 .85 1.00
I6 .88 .91 .95 .87 .85 1.00
Item 5

Item 6

1/31/2020
.90 41
Internal Consistency Reliability
Average item-total correlation

1/31/2020 42
Internal Consistency Reliability
Average item-total correlation

Test

1/31/2020 43
 Internal Consistency Reliability
Average item-total correlation

Item 1

Item 2

Item 3
Test
Item 4

Item 5

Item 6
1/31/2020 44
 Internal Consistency Reliability
Average item-total correlation

Item 1
I1 I2 I3 I4 I5 I6
Item 2
I1 1.00
I2 .89 1.00
Item 3 I3 .91 .92 1.00
Test I4 .88 .93 .95 1.00
Item 4 I5 .84 .86 .92 .85 1.00
I6 .88 .91 .95 .87 .85 1.00
Total .84 .88 .86 .87 .83 .82 1.00
Item 5

Item 6
1/31/2020 45
 Internal Consistency Reliability
Average item-total correlation

Item 1
I1 I2 I3 I4 I5 I6
Item 2
I1 1.00
I2 .89 1.00
Item 3 I3 .91 .92 1.00
Test I4 .88 .93 .95 1.00
Item 4 I5 .84 .86 .92 .85 1.00
I6 .88 .91 .95 .87 .85 1.00
Total .84 .88 .86 .87 .83 .82 1.00
Item 5

Item 6
1/31/2020 46
.85
Internal Consistency Reliability
Split-half correlations

1/31/2020 47
Internal Consistency Reliability
Split-half correlations

Test

1/31/2020 48
 Internal Consistency Reliability
Split-half correlations

Item 1

Item 2

Item 3
Test
Item 4

Item 5

Item 6
1/31/2020 49
 Internal Consistency Reliability
Split-half correlations
Item 1

Item 2 Item 1 Item 3 Item 4

Item 3
Test
Item 4

Item 5

Item 6

1/31/2020 50
 Internal Consistency Reliability
Split-half correlations
Item 1

Item 2 Item 1 Item 3 Item 4

Item 3
Test
Item 4

Item 5

Item 2 Item 5 Item 6

Item 6

1/31/2020 51
 Internal Consistency Reliability
Split-half correlations
Item 1

Item 2 Item 1 Item 3 Item 4

Item 3
Test
Item 4 .87
Item 5

Item 2 Item 5 Item 6

Item 6

1/31/2020 52
Internal Consistency Reliability
Cronbach’s alpha ()

1/31/2020 53
Internal Consistency Reliability
Cronbach’s alpha ()

Test

1/31/2020 54
 Internal Consistency Reliability
Cronbach’s alpha ()

Item 1

Item 2

Item 3
Test
Item 4

Item 5

Item 6
1/31/2020 55
 Internal Consistency Reliability
Cronbach’s alpha ()

Item 1
item 1 item 3 item 4 item 1 item 3 item 4 item 1 item 3 item 4

.87 .87 .87

Item 2
item 2 item 5 item 6 item 2 item 5 item 6 item 2 item 5 item 6

Item 3
Test
Item 4

Item 5

Item 6
1/31/2020 56
 Internal Consistency Reliability
Cronbach’s alpha ()

Item 1
item 1 item 3 item 4 item 1 item 3 item 4 item 1 item 3 item 4

.87 .87 .87

Item 2
item 2 item 5 item 6 item 2 item 5 item 6 item 2 item 5 item 6

Item 3
SH1 .87
Test
SH2 .85
Item 4 SH3 .91
SH4 .83
Item 5 SH5 .86
...
SHn .85
Item 6
1/31/2020 57
 Internal Consistency Reliability
Cronbach’s alpha ()

Item 1
item 1 item 3 item 4 item 1 item 3 item 4 item 1 item 3 item 4

.87 .87 .87

Item 2
item 2 item 5 item 6 item 2 item 5 item 6 item 2 item 5 item 6

Item 3
SH1 .87
Test
SH2 .85
Item 4 SH3 .91
SH4 .83
Item 5 SH5 .86
...
SHn .85
Item 6
 = .85
1/31/2020 58
 Internal Consistency Reliability
Cronbach’s alpha ()

Item 1
item 1 item 3 item 4 item 1 item 3 item 4 item 1 item 3 item 4

.87 .87 .87

Item 2
item 2 item 5 item 6 item 2 item 5 item 6 item 2 item 5 item 6

Item 3
SH1 .87
Test
SH2 .85
Item 4 SH3 .91 Like the average of all
SH4 .83 possible split-half
SH5 .86 correlations
Item 5
...
SHn .85
Item 6
 = .85
1/31/2020 59
 References
• http://www.cpp.edu/~smemerson/nbu-program
eval/trochimppp/Part%203/Types%20of%20Reliability.ppt
• Bee Bornheimer, Robin Fitzpatrick, Sarah Lehmann, Matt Pierce, and Maureen
Whalen. Reliability and Validity in Research.2008
• Prof. Rodney Ehrlich. Variable definition and quality control of measurements .
Teaching Registrars Research Methods .
• Research Methods Knowledge base. Available on:
http://www.socialresearchmethods.net/kb/.

1/31/2020 60
 THANK YOU!!!!!!!!!!!!!!!!!

1/31/2020 61