Martinez, Maria Regina J. OrCom 109.

2 (TFI)
2008-18861 Prof. Rafael Villar

INSTRUMENT VALIDITY AND RELIABILITY

Validity
 A true measure
 The ability of measurement procedures to produce empirical data consistent with the theoretical meaning of the
concept to be measured
 An indicator may be valid for a particular purpose and definition, but less valid or invalid for others
 We can never have absolute validity but some measures are more valid than others

Four Types of Measurement Validity

1. Face Validity
- Easiest type to achieve and most basic kind of validity
- A scale is accepted as valid if it looks or sounds valid to the researcher
- A judgement by the scientific community that the indicator really measures the construct
- Weak evidence because it is basically a judgement call, but does not necessarily mean wrong
- Quality of face validity assessment can be improved considerably by making it more systematic
- Question answered:
On the face of it, do people believe that the definition and method of measurement fit?

2. Content Validity
- Captures the entire meaning
- Measures should sample or represent all ideas or areas in a conceptual space
- Comparing the items in a measure with items which could have been used permits assessing its content validity
- Question answered:
Is the full content of a definition represented in a measure?
- Three steps:
1) Specify the content in a construct’s definition
2) Sample from all areas of the definition
3) Develop an indicator that taps all of the various parts of the definition

3. Criterion Validity
- Agrees with an external source
- Uses some standard or criterion that is known to indicate a construct accurately
- The validity of an indicator is verified by comparing it with another measure of the same construct in which a
researcher has confidence
- Two sub-types:
a. Concurrent
 - An indicator must be associated with a pre-existing behavior just that is to be judged to be valid
- Ability to distinguish between groups that it should theoretically be able to distinguish between
b. Predictive
- An indicator predicts future events that are logically related to a construct
- Measure can correctly predict something that theoretically it should be able to predict
4. Construct Validity
- Multiple indicators are consistent
- Requires a definition with clearly specified conceptual boundaries
- Question answered:
If the measurement is valid, do the various indicators operate in a consistent manner?
- Two sub-types:
a. Convergent
- “Alike ones are similar”
- Multiple measures of the same construct hang together or operate in similar ways
b. Discriminant (divergent)
- “Different ones differ”
- Indicators of one construct hang together or converge, but also diverge or are negatively associated with
opposing constructs

Reliability
 It is a dependable measure
 Response is similar and predictable in different times and conditions
 The method of conducting a study or the results from it can be reproduced or replicated by other researchers
 Stability and internal consistency
 Reliability is inversely related to random error

Four Types of Reliability

1. Inter-Rater or Inter-Observer Reliability
- Assesses the degree to which different observers give consistent estimates of the same phenomenon
2. Test-Retest Reliability
- Measures the consistency of a measure from one time to another
3. Inter-Method Reliability
- Considers the consistency of the results of two tests constructed in the same way from the same content
domain
- The variation in measurements with different methods, but taken by the same person
4. Internal Consistency Reliability
- Calculates the consistency of results across items within a test

Improving Reliability
- Item analysis – most effective way to increase reliability
- Consists of computation of item difficulties and item discrimination indices
 - Items that are too difficult, too easy, and/or have near-zero or negative discrimination are replaced with better items

Relationship Between Validity and Reliability

 Reliability is necessary for and easier to achieve than validity, although not a guarantee that a measure will be
valid
 A measure can be reliable but invalid
 Usually complementary concepts, but sometimes conflict with each other
 Sometimes as validity increases, reliability is more difficult to attain (vice versa)
 Reliability is easiest to achieve when measure is precise and observable

Sources are stated on the cut-and-paste.

Tables and examples will be discussed in class.