EAA492/6: FINAL YEAR PROJECT

DEVELOPING QUESTIONNAIRES
PART 2
1

AHMAD SHUKRI YAHAYA

ENGINEERING CAMPUS
USM
 CONTENTS
2

 Reliability And Validity

 Sample Size Determination
 Sampling Designs
 Descriptive And Inferential Statistics
 Pilot Study
 CHOICE OF QUESTIONNAIRES
3

 Adopt the questionnaire based on previous studies

 Adapt the questionnaire based on previous studies

 Create a NEW questionnaire

 RELIABILITY AND VALIDITY
4

 Validity can be defined as the extent to which any

measuring instrument measures what it is intended
to measure.

 Reliabilityconcerns the extent to which an

experiment, test or any measuring procedure yields
the same results on repeated trials.
 VALIDITY
5

 Validity is the extent to which any measuring

instrument measures what it is intended to measure.
 Validity is about interpretation of data arising from a
specified procedure. It is not a test!
 Thus, validity is not about the measuring instrument
itself but the measuring instrument in relation to the
purpose for which it is being used.
 VALIDITY
6

 Three types of validity:

(1) Content Validity
(2) Criterion-Related Validity
(3) Construct Validity
 Content Validity
7

 Content validity depends on the extent to which an

empirical measurement reflects a specific domain of
content.
Example: A test in arithmetical operations would not be
valid if the test problems focused only on addition and
neglecting subtraction, multiplication and division.
 Thus a researcher must be able to specify the full domain
of content that is relevant to the particular measurement
situation. Example: We must specify all the words that a
standard four student should know how to spell. Choose
at random the number of words that should be sampled.
Then, they must be put in a form that is testable.
 Content Validity – Some limitations
8

 The process of determining the domain of the

content is more difficult and complex when dealing
with the abstract concepts typically found in the
social sciences.
 There is no agreed upon criterion for determining
the extent to which a measure has attained content
validity.
 Thus, a measure can only be considered as strongly
or weakly valid (i.e., the alternative is not between
fully valid or fully invalid measures).
 Criterion-Related Validity
9

 Also known as Predictive Validity

 Has the closest relationship to what is meant by the term
validity
 Definition: Is an issue when the purpose is to use an
instrument to estimate some important form of
behaviour that is external to the measuring instrument
itself, the latter being referred to as the criterion
 Example: We assess the validity of college board
examination by showing they accurately predict how well
high school college seniors will do in college instruction
 Criterion-Related Validity
10

 Example: We validate a written driver’s test by

showing that it accurately predicts how well some
group of persons can operate an automobile.
 The indicator between the test and the criterion is
usually estimated by the size of the correlation.
 Criterion-Related Validity
11

 Have been used mainly in psychology and education.

 Should be used in any situation or area of scientific
enquiry in which it makes sense to correlate scores
obtained on a given test with performance on a
particular criterion or set of relevant criteria.
 For some cases, criterion-related validity cannot be
used because we cannot determine the relevant
criterion variables.
 Construct validity
12

 It is concerned with the extent to which a particular measure

relates to other measures consistent with theoretically derived
hypotheses concerning the concepts (or constructs) that are
being measured.
 Example: Suppose a researcher wanted to evaluate the
construct validity of a particular measure of self-esteem – say
Rosenberg’s self-esteem scale. Theoretically, Rosenberg has
argued that a student’s level of self-esteem is positively related
to participation in school activities.
 Determine correlation between Rosenberg’s self-esteem
scale to a group of students and their extent of involvement in
school activities. If correlation is positive and substantial,
then it supports one piece of evidence on the validity of
Rosenberg’s self-esteem scale.
 Construct validity
13

 Involves three steps:

(1) Theoretical relationships between the concepts
must be specified.
(2) Empirical relationships between the measures of
the concepts must be examined.
(3) Empirical evidence must be interpreted in terms
of how it clarifies the construct validity of a
particular measure.
 RELIABILITY
14

 Reliability concerns the degree to which results are

consistent across repeated measures

 Basic formulation of measurements

where X is the observed score, t is the true score

and e is the random error
 RELIABILITY
15

 Assumptions:
1.
2.
3.
4.

Note : For (3), it is assumed that two sets of measurements

are observed for a single person for a single variable.
 RELIABILITY
16
Therefore,

From Assumption (1),

This result is true for repeated measurements of a single variable

for a single person.
 RELIABILITY
17

 Reliability refers to the consistency of repeated

measurements across persons rather than within a
single person.
 Thus, look at the variance of the measurement.
 Hence
 RELIABILITY
18

 Thus the ratio of true to observed variance is called

the reliability of X as a measure of T.

 Reliability can also be expressed as

 RELIABILITY
19

 The estimate of a measure’s reliability can be

obtained by correlating parallel measurements.
 Two measurements (X and X´) are defined as
parallel if they have identical true scores and equal
variances as shown below:
 RELIABILITY
20

Thus,

Thus it follows that the estimate of reliability is simply the correlation

between parallel measures.
 RELIABILITY
21

 There are four basic methods to estimate the

reliability of empirical measurements namely the
retest method, the alternative-form method, the
split-halves method and the internal consistency
method
 The range of values for the reliability method is from
0 to 1.
 Values near 1 show good reliability.
 Usually, if the value is more than 0.7 than the
method is reliable.
 RELIABILITY
22

 In SPSS, the reliability analysis is obtained from the

following commands:
ANALYZE – SCALE – RELIABILITY ANALYSIS
 Under reliability, there are five different types of
methods namely
(1) The alpha Cronbach’s method
(2) The split-half method
(3) The Guttman method
(4) The parallel method
(5) The strict parallel method
 The Retest Method
23

 The easiest method.

 Suppose that a set of questionnaires or tests are
given to some respondents. Then if the same set of
questionnaires or tests are given to the same
respondents after some specified time period, then
this is known as the retest method.

 The interval between the two tests are usually taken

to be from two to four weeks.
 The Retest Method
24

 The equations for the two tests are as follows:

X 1  X t  t
andX 2  X t  2

 Assumptions:
V 1   V 2 
(i)
1 , 2   0
(ii)
 Thus
 x    X  X
1 2
 The Retest Method
25

 Weaknesses of this method

1. Researches usually cannot do more than one tests
2. The reaction of the respondents about certain surveys.
Example: If a respondent is being interviewed about
whether he/she will vote in a coming election at time 1, the
respondent might make a decision at time 2 and will actually
vote at time 3 due to the fact that he/she was sensitized to
the election through the interview.
 The alternative form method
26

 The most frequently used method in the field of

education
 Similar to the retest method as it requires two sets of
tests which is given to the same respondents
 Different from the retest method in that an
alternative form of the questions are given
 The two sets of questions must measure the same
thing. Example: If two tests are designed to
measure the understanding of mathematical
operators using 20 questions for each test then the
sets of questions must be of equal difficulty
 The alternative form method
27

 Superior than the retest method

 Weakness of this method is to design questionnaires

or tests which are of equal level.
 The split-half method
28

 Suppose there are N questions in a questionnaire

 These questions are split into two equal halves each
having N/2 questions.
 Split can be done arbitrarily.
Example: Can choose the first questions and the
other being the last questions or we can choose the
even numbered questions for the first half and the
odd numbered questions as the second half.
 The value of the reliability measures will be different
for different splits.
 The split-half method
29

 The Spearman-Brown prophecy formula for

measuring reliability is given by :
2 xx
xx 
1  xx

xx is the reliability for the whole sample

xx is the correlation between the two split-half

 Internal consistency method
30

 This method require only a single test on the sample

and is usually known as the internal consistency
method.
 The most popular of the internal consistency method
is the Cronbach’s alpha method which is given by
N

[1   ( N  1)]

N is equal to the number of questions

 is the mean correlation between each questions

 CONTENTS
31

 Reliability And Validity

 Sample Size Determination
 Sampling Designs
 Descriptive And Inferential Statistics
 Pilot Study
 Sample Size Determination
32

Sample
Population

Sample is chosen at random from a population

 Sample Size Determination
33

 Sample size depends on the budget and degree of

confidence required.
 Smaller samples are more likely to be different from
the population than larger ones. So smaller samples
have more sampling error and lower reliability.
Sample Size

Sampling error

Sample Reliability
 Sample Size Determination
34

 Krejcie, R.V. and Morgan, D.W. (1970), Determining

Sample Size For Research Activities, Educational
And Psychological Measurement, 30, 607-610.
 The following Table 1 is from this paper.
 Sample Size Determination
35

N S N S N S
140 103 700 248 10000 370
150 108 750 254 15000 375
160 113 800 260 20000 377
170 118 850 265 30000 379
180 123 900 269 40000 380
190 127 950 274 50000 381
200 132 1000 278 75000 382
210 136 1100 285 1000000 384

N is the population size; S is sample size

 CONTENTS
36

 Reliability And Validity

 Sample Size Determination
 Sampling Designs
 Descriptive And Inferential Statistics
 Pilot Study
 SAMPLING
37

 Foundation of a good sample survey is the sample.

 A sample is some part of a larger body specially
selected to represent the population.
 Sampling is the process by which it is done.
 Samples must be representative of the population.
 PROBABILITY AND NONPROBABILITY
SAMPLING
38

 Probability sampling is a process of sample

selection in which elements are chosen by chance
procedures and with known probabilities of
selection.
 Nonprobability sampling includes all methods
in which units are not selected by chance procedures
or with known probabilities of selection.
 NONPROBABILITY SAMPLING
39

 Haphazard sampling: Samples are made up of

individuals casually met or conveniently available such
as students enrolled in a class or people passing by on a
street corner. Cannot make generalization beyond the
collections themselves and are seldom of scientific
interest. Also known as convenience sampling.
 Judgmental or purposive sampling: Sample
elements are chosen from the population by interviewers
using their own discretion about which informants are
“typical” or “representative”. Results of such sampling
procedure can be very good, if the interviewers intuition
or judgment is sound.
 NONPROBABILITY SAMPLING
40

 Quota sampling: Process of selection in which the

element are chosen by interviewers using
prearranged categories of sample elements to obtain
a predetermined number of cases in each category.
 Expert sampling: Elements are chosen on the
basis of informed opinion that they are
representative of the population in question.
Example: A specialist on secondary education may
decide that four schools across the country
adequately represent the range of variation seen in
teaching methods.
 PROBABILITY SAMPLING
41

 Simple Random Sampling (srs): Each

population member has the same probability of
appearing in the sample.
 Sample size: Depends on the objective of survey.
 Assume that we need to estimate the population
mean, by using a srs mean and restricting to an
acceptable level the probability that the absolute
difference between the population mean and the
sample mean is greater than some specified value.
 Simple Random Sampling
42

Then we have

for some given d and α

Thus,

where
 Determination of S2 in SRS
43

 From pilot studies

 From previous surveys
 From a preliminary sample
 PROBABILITY SAMPLING
44

 Systematic sampling: Method of selecting units

from a list through the application of a selection
interval, I, so that every
Ith unit on the list, following a random start , is
included in the sample.
 Sample size: Depends on the objective of survey.
 Assume that we need to estimate the population
mean, by using a sample mean restricting to an
acceptable level the probability that the absolute
difference between the population mean and the
sample mean is greater than some specified value.
 Systematic Sampling
45

Then we have

for some given d and α

Thus,

where
 PROBABILITY SAMPLING
46

 Stratified (simple) random sampling: Technique

where a population can be conveniently partitioned into
a set of sub-populations (strata). Such a population is
said to be stratified. Within a strata, simple random
sampling method is used to determine the sample.

 Cluster sampling: Sometimes a finite population may

consist of a large number of groups of individuals, e.g. of
households in a city. This is a special form of
stratification (many strata of rather small size) and is
referred as clusters. Draw a cluster sample as a srs of the
clusters. If all the members of the sampled clusters are
obtained , this is known as one-stage cluster
sampling.
 CONTENTS
47

 Reliability And Validity

 Sample Size Determination
 Sampling Designs
 Descriptive And Inferential Statistics
 Pilot Study
 Descriptive and Inferential Statistics
48

 Descriptive Statistics- used to describe the data where

data are presented in the form of tables, charts or
summarization by means of percentiles and standard
deviation
 Measures of locations: mean, median, mode
 Measures of spread: standard deviation, variance, range.
 Plots such bar chart, pie chart, histogram, Box and
Whiskers plot.
 Not enough just to do this in your FYP!!!
Descriptive and Inferential Statistics
49

Sample
Population

Sample is chosen at random from a population

Descriptive and Inferential Statistics
50

 Inferential statistics
- Process of drawing information from
sampled observations of a population and
making conclusions about the population.
-Two-prong approach.
(1) Sampling must be representative of
population
(2) Correct conclusions made of
population
Descriptive and Inferential Statistics
51

 Inferential statistics
- t-tests
- Analysis of variance tests
- Chi-square tests
- Regression models
 Students must do some inferential statistics.
 CONTENTS
52

 Reliability And Validity

 Sample Size Determination
 Sampling Designs
 Descriptive And Inferential Statistics
 Pilot Study
 Pilot Study
53

 Last major stage of survey work before data collection

stage.
 Designed to find any problems with the data collection
process such as:
- Poor introduction and instructions to questionnaire
- Unclear or undefined terms
- Unclear or ambiguous response task
- Too many “don’t know” responses
- Biased or offensive questions
- and so on.
 Check reliability
 Pilot Study
54

 Choose potential respondents to complete a

questionnaire
 Other approaches are:
- Behaviour coding: Investigator watches the
respondent and/or interviewer complete the
questionnaire or observes the behaviour after it has
been recorded on tape.
- Cognitive interview: Respondents are asked to
think aloud while completing the survey and to
describe everything that comes to mind while
arriving at an answer.
 Pilot Study
55

 Interviewer evaluation: Interviewers are asked to

code question characteristics and respondent
behaviour.
 Respondent evaluation: Respondents are asked
to rate and/or comment about the questions.
 Expert panels: Experts in survey research can be
asked to review a questionnaire and identify
potential problems.
 Guidelines
56

 Sample size: At least 25 samples. For behaviour and

cognitive interview, sample size can be reduced to about
12 samples.
 Sample composition: Should be similar to that of the
survey.
 Number of pretests: One pretest is adequate but not
always recommended.
 Data collection time: For interviews, can allow 50%
longer than the projected interview.
 Statistical analysis: Can be done if data is more than 25.
 Number of identified problems: Will definitely find
problems.
 Measure the reliability of the questionnaire
 SOME COMMENTS
57

 Before the pilot study, the following must be

followed:
(1) For questionnaires that are adopted and adapted
(with less than 20% change), content validity need
not be checked.
(2) For NEW and adapted (more than 20%)
questionnaires, content validity must be done with at
least three experts.
 SOME COMMENTS
58

 During the pilot study, for NEW questionnaires,

(1)the number of sample size must be more than 100
(2)factor analysis must be carried out.
 SUMMARY
59

 FYP report must include

(1) Validity – Content Validity
(2) Reliability
(3) Pilot Study
(4) Descriptive Statistics
(5) Inferential Statistics
THANK YOU
60