RELIABILITY AND VALIDITY

Quality of
Data

1
 PRIMARY SOURCES
A prim ary s o urce is 
an original object or
document -- the raw
material or first-hand
.information
Eye witness accounts 
are also a primary
source, but are less
reliable than a
. document
 A PRIMARY SOURCE IS AN ORIGINAL OBJECT OR
DOCUMENT -- THE RAW MATERIAL OR FIRST-HAND
. INFORMATION
historical and legal
documents
results of an
experiment
statistical data
pieces of creative
writing
art objects
eye witness accounts
 SECONDARY SOURCES

A secondary source is something

written about a primary source.
 You can think of secondary
sources as second-hand
information. If I tell you something,
I am the primary source. If you tell
someone else what I told you, you
are the secondhand source.
 SECONDARY SOURCES

Se co nda ry s o urce s

in clu de :
comments on
interpretations of
discussions about
the original material
 SECONDARY SOURCES

 Secondary source materials can be:

 articles in newspapers
 popular magazines
 book or movie reviews
 articles found in scholarly journals that evaluate or
criticize someone else's original research
WHAT IS THE SOURCE OF THE
INFORMATION BEING
? CONSIDERED
Pub lic atio n
-Date - is the information current, or does it
need to be current?
-Reputation of publication - is the source well
known and trustworthy?
-Kind of publication - is it a scientific report,  
eye-witness account, a work of fiction?
 QUANTITATIVE DATA

Reliability
Validity
 External
 Internal

Generalizability

8
 RELIABILITY

Implies that the same data would have been

collected each time over repeated tests/
observations.
Would a particular technique (or survey
question) yield the same result each time?
 “Did you go to church last week?” vs. “How many
times have you been to church in your life?”
Reliability does not ensure accuracy.

9
 RELIABILITY

Problem if interpret questions differently

Poorly worded questions
Inconsistent coding: coding errors as with open-
ended questions
Lack of definition of key terms

10
 RELIABILITY

Poorly worded: Does the library have adequate

facilities and equipment for physically disabled
students
Better: Can patrons in wheelchairs retrieve books
from the browsing collection?

11
 RELIABILITY (INDICATORS)

Pretest
Repeat question(s)
Test/retest
Split half and Parallel
Interscore or scorer

12
 QUALITATIVE RELIABILITY

Researcher is the “instrument”- how to test for

reliability?
 Provide details of method, and abundance of evidence
 Provide evidence of qualifications as observer
 Make assumptions (and possible biases) clear
 State research questions clearly
 Use early stages of study to generate focus
 Observe for an adequate period of time, across a full range of
activities
 Collect data from multiple sources
 Save data for reanalysis

13
 Perhaps the best way to think of the reliability of information
is to think of it as existing on a scale, rather than falling into
the categories of reliable or unreliable. Information can range
from very unreliable to highly reliable and also anywhere in
between.
 VALIDITY

Validity in qualitative research means

“appropriateness” of the tools, processes, and
data
A term to describe a measure that accurately
reflects the concept it is intended to measure.
Which is a more “valid” indicator of intelligence-
an IQ score, or number of hours spent studying?
Ultimate validity cannot be proven, but can be
supported by face, internal, and external
measures.
 Babbie, E.

15
 TYPES OF VALIDITY

 Face validity: The quality of an indicator/ question/ test that

makes it a reasonable measure of a variable.
 Church attendance is an indication of religiosity.
 Number of grievances filed is an indicator of worker morale

16
 INTERNAL VALIDITY

Approximate truth about inferences regarding causal

relationships
Typically applied to studies using inferential
statistics (i.e. quantitative measures) than
descriptive or observation studies.
Especially useful for studies assessing affects of
programs
Only applicable to the study in question- not
generalizable. Why not?
Key question: Whether observed changes can be
attributed to your program (the cause) and NOT other
possible impacts/ causes.

17
INTERNAL VALIDITY

18
 HISTORY MATURATION
TESTING

DIRECTION
OF CAUSALITY

-INSTRUMENT
ATION
THREATS TO
TYPE 1 AND VALIDITY AND
TYPE 2 RELIABILITY
ERRORS

EXPERIMENTAL
MORTALITY
-OPERATIONAL
IZATION
-CONTAMIN
ATION REACTIVITY
 INTERNAL VALIDITY

 H i s t o r y o r s p e c i f i c e v e n t s : raises the issue that some variable other

than the independent variable accounted for the change in the dependent
variable. E.G.: the length of time between conducting the pretest and
posttest may have a detrimental effect.
 M a t u r a t i o n : the change results from biological or psychological
processes, which occurred over time, and not from the treatment itself.
Maturation becomes more a concern the longer the period between the
pretest and posttest
 P r e t e s t i n g : may affect the dependent variable. Pretesting may alert
participants or educate them about the topic under investigation.
Therefore if subjects are administered a posttest, their performance may
reflect a marked improvement
 M e a s u r i n g i n s t r u m e n t s o r o b s e r v a t i o n a l t e c h n i q u e s : These—not the
treatment—may account for the change in the dependent variable.
Further, the validity of study findings may have been influenced by the
fact that the evaluators as observers, raters, graders, interviewers, and
coders gained experience, became tired, obtained a more complete
understanding of the project, or eased their expectations of test subjects
20
 INTERNAL VALIDITY

A n o n r a n d o m a s si g n men t o f su b j e c t s to groups
may signify that the groups were dissimilar from the
beginning. Therefore any change might be attributed
to the differential selection of subjects, rather than
the actual treatment.
Sta ti s ti c a l re g re s s i o n refers to the tendency for
extreme scores to regress or move toward the
common mean of subsequent measures. The
assignment of subjects to a particular test group on
the basis of extreme views may affect study findings.

21
 INTERNAL VALIDITY

M o r t a l i t y refers to the possibility that some subjects

may have dropped out of the study after completion
of the pretest but before the administration of the
posttest. In such instances, every effort should be
made to identify any common patterns or
characteristics to ensure that any difference between
a group’s pretest and posttest scores cannot be
attributed to the loss of subjects.
I n t e r ac t i o n refers to the fact that more than one of
the previous threats might be in play. This is
especially likely in those cases where subjects were
not randomly assigned to groups and the evaluation
was based on existing, intact groups.
22
 EXTERNAL VALIDITY

The approximate truth of generalizations drawn from

a study.
The degree to which conclusions drawn from your
study sample would hold true to other persons in
other places at other times

23
EXTERNAL VALIDITY

24
 EXTERNAL VALIDITY

 Example: institutions of higher education in Massachusetts:

control, highest degree offered, and some characteristics of
library (staff number, budget, and volume number)
 Return rate?
 Do respondents differ from non-respondents as a group?

25
 VALIDITY

 Content validity (for achievement test): How well does the test
sample what the students learned? How well does a
standardized test cover what was taught in the information
literacy program?

26
 VALIDITY

 Criterion-related (predictive) (attitude test to predict

performance in a library skills program): Who well does the
test predict achievement for college freshmen?
 Criterion-related (diagnostic ): How well does the test diagnose
current problems with library use?

27
 QUALITATIVE VALIDITY

Depends upon reliablity. Like reliability, asserted by

documenting steps
 Triangulation- data from different sources/ methods
 Full documentation of data- “chain of evidence”
 Logical connections between data and conclusions
 Conscious and deliberate inclusion of data that might not
support thesis
 Preparedness to entertain alternatives
 Self-reflection, acknowledgement of own biases
 Review of preliminary reports by objective observers
 Awareness of limitations

28
 GENERALIZABILITY

• Generalizability refers to the extent to which research

findings are applicable to other populations or samples.
• It involves the usefulness of one set of findings in explaining
other similar situations.
• It is sometimes equated with terms of transferability and
external validity.
• Generalizability of a finding (broadly speaking) is whether or
not the results will hold for all (or at least most) combinations
of the elements of external validity
 GENERALIZABILITY OF RESEARCH
IN QUALITATIVE STUDIES
Generalizations are made, or Theories to Past Experience
And Literature

Researcher Looks for Broad Patterns, Generalizations, or

Theories from Themes or Categories

Researcher Analyzes Data to Form Themes

Or Categories

Researcher Asks Open-Ended Questions of Participants

Or Records Field Notes

Researcher Gathers Information

30
 CHARACTERISTICS OF A “GOOD”
QUALITATIVE STUDY

1. Multiple forms of data are gathered and analyzed. The

“data set” is visually summarized (creativity without
gimmickry is encouraged!).
2. The study is framed within the appropriate tradition.
3. The researcher begins with a single focus (not a causal
or comparative approach).
4. The study provides a detailed description of the
method, enough so that it could be replicated
(approach this as you would if you were writing a
screenplay).

31
 CHARACTERISTICS OF A “GOOD”
QUALITATIVE STUDY

5. The writing is persuasive (i.e., verisimilitude).

6. Analyses are layered.
7. The writing is rich with detail and brings the story to
life.
8. The assumptions and characteristics of qualitative
inquiry are kept intact (e.g., evolving design,
presentation of multiple perspectives, the full
engagement of the researcher is present,
trustworthiness, etc.)

32
 . . . CARS is an acronym for

, Credibility
, Accuracy
, Reasonableness
&
Support

CARS
 CARS

Credibility
Check the URL (.gov or .edu)
Check the sponsor (a univ or goverment or medical
agency)
Can you identify the author?
Purpose (are they selling or informing or just talking?)
 CARS

Accuracy
Correctness
Up to date (for what it is)
Spelling
Grammar mistakes
 CARS

Reasonableness
Tone (is it angry or whiny or professional?)
Bias (does it lean more toward one side?)
Logic (does it make sense?)
 CARS

Support

Sponsored
Cited (research evidence provided)
Contact information