DESIGNING AND VALIDATING A SURVEY QUESTIONNAIRE ON

STUDENTS’ RESEARCH COMPETENCE AND APPRECIATION

Soren R. Sanchez, MAED

University of Cebu, Cebu City, Philippines
sorensanchez81@gmail.com

Frankie M. Mendez, MAED

Mabolo Elementary School, Cebu City, Philippines

A B S T R A C T

Like Mathematics, research is one of those subjects students often

hold an adverse connotation and attitude against it. It is being generalized as
difficult and time-consuming. Students usually experienced doing research
only when they reached tertiary education before but not until the
implementation of the K to 12 Curriculum, which the Philippine educational
system has just shifted with. One important feature of this new curriculum is
the 2-year extension of the basic education- the senior high school. Likewise,
the presence of research curricular offerings given to the senior high school
students is also what sets it apart from the old curriculum. Hence, it is
expected that the college students who are K to 12 products are possess
better skills and appreciation towards research. This study intends to develop
and validate a survey questionnaire which aims to determine students’ level
research competence and appreciation. More specifically, it intends to
determine the reliability of the instrument and establish evidences for the
instrument’s validity. The use of Cronbach’s Alpha enabled the instrument
designers to determine the reliability of the instrument. Likewise, the
stakeholders, content experts such as the research directors and instructors
and the potential respondents were consulted to seek feedback for
consideration and revision. Results revealed that the instrument has
collected evidences for face validity, content validity, criterion validity and
construct validity. The construct validity was determined through Exploratory
Factor Analysis (EFA). Hence, the research instrument has proven that it is
reliable and valid and is now ready to be administered to the real
respondents of the final study.

Keywords: Survey Questionnaire, Research Competence, Research Appreciation

1
Rationale
Research is one of the integral subjects offered in higher education. Gomez and
Panaligan (2013) defined research as the systematic way of creating, discovering and
innovating new knowledge. However, a lot of undergraduate students, as novice
researchers, stigmatize research and other research-related courses with an adverse
attitudes and feelings (Papanastasiou, 2005; Ellis and Levy, 2010). One factor that has
caused this is the low research competence of the neophyte researchers. Research
competence refers to the needed experiences and skills to write and conduct research
(Malari and Santiago, 2013; Swank and Lambie, 2016). These skills and experiences
are ameliorated through exposure such as schooling and attending seminars/
workshops.

The dynamics in doing research makes it difficult for students to conduct

research from specifying the problem (Ellis and Levy, 2008), anchoring the problem to a
theory (Levy and Ellis, 2006), drafting the literature review and planning what method
and design best suit to the nature of study (Ellis and Levy, 2009; Ellis and Levy, 2010).
In addition, the kind of research mentorship plays a vital role in enhancing the
undergraduate’s level of research competence (Norem & Lonneman-Doroff, 2009;
Lambie & Vaccaro, 2011; Hollingsworth & Fassinger, 2002; Borders et al., 2012).

In the Philippine context, the country’s educational system has shifted from the
old curriculum to the new curriculum otherwise known as the K to 12 Curriculum. One of
the salient features of such curriculum is the procurement of two years for senior high
school, wherein they receive two or more research subjects depending on the track and
strand they opt to pursue. As compared to the old curriculum, students experience
research only once they reach the tertiary level, not on the secondary level. Vivid
enough, the research competence and appreciation of the K to 12 Curriculum product
tertiary students are quite different from those of the old curriculum. It is therefore
expected that their level of competence and appreciation in research writing are higher
than the old curriculum product.

Furthermore, there had been a number of researches conducted related to

research skills of students. Most them were focusing on measuring their research
attitudes and competence (Meerah et. al, 2012; Swank and Lambie, 2016; Şahan &
Tarhan, 2015), and research competence and satisfaction (Gomez and Panaligan,
2013). However, only few of these related studies gave focus on designing and
validating an instrument to measure the undergraduate’s research competence and
appreciation; a kind of research tool which can help the researcher and teacher identify
the portion of doing research where students encounter much difficulty. With this,

2
educators may develop teaching approaches and/or programs to intensify the teaching
of such portion in doing research; overall making the learning, writing and conducting
research easier.

The researchers wanted to contribute to the knowledge-base of doing research

and by developing a survey questionnaire that aims to determine students’ the level of
research competence, the part of doing research where students find the most difficult
and the level of extent on how much they value research.

Objectives of the Study

This study gears to develop and validate an instrument that determines the
research competence and appreciation of college students. Specifically, this study
intends to determine the reliability of the instrument and establish evidences for the
instrument’s validity.

Methodology
This study is quantitative by nature, a descriptive-survey method through a
researcher-made questionnaire measuring 2 constructs namely: (1) Research
Competence (44 items) and (2) Research Appreciation (15 items). The first construct
was subdivided into 3 areas namely: (1) Research Dynamic Competencies (15 items),
(2) Writing Competencies (23 items), and (3) technical competencies (6 items). They
were administered to the randomly selected 100 first year college education students
who are product of the k to 12 Curriculum.

For the first construct, a rubrics–type scale that ranges from 1 (No
Experience/Proficiency), 2 (fundamental awareness/ basic knowledge), 3 (novice
learner), 4 (intermediate proficiency), 5 (advanced proficiency) to 6 (Expert Proficiency)
was utilized, while for the second construct, a semantic differential was used where the
respondents describe their appreciation towards research by checking the boxes from 1
to 6 where 1 is the lowest and 6 is the highest. The survey items in both constructs were
made in a self-report format..

The study was conducted in the University of Cebu, College of Teacher

Education Department– Main Campus, Cebu City, Philippines, School Year 2019 -
2020. This is a private non-sectarian school its Bachelor in Elementary Education
(BEED) Program is Level 3 and the Bachelor in Secondary Education (BSED) Programs
are Level 2 accredited by the Philippine Association of Colleges and Universities
Commission on Accreditation (PACUCOA), a private accrediting agency which gives a
formal recognition to an educational institution by attesting that its academic program
maintains excellent standards in its educational operations, in the context of its aims
and objectives.

3
 In the development of the Students’ Research Competence and Appreciation
instrument, the researchers made use of the process of instrument construction by
Colton and Covert (2007) as shown in Figure 1.

I
Identify the purpose
and focus of the
study Obtain feedback from
Administer the stakeholders to
instruments, analyse clarify the purpose and
focus
and report results

Identify the type of

Pilot test and revise instrument and
methodology to use for
prior to final
the data collection and
administration
measurement
,easurement
Revise instruments
based on feedback; Begin to formulate
prepare for pilot questions or items
Pretest items and
testing
preliminary draft (With
Content experts,
stakeholders and
Potential respondents)

Fig. 1. Steps in the instrument construction process. Adapted from “Designing

and Constructing Instruments for Social Research and Evaluation,” by David
Colton and Robert Covert, 2007, p.18.

The Process of Instrument Development

As specified by Colton and Covert’s (2007) comprehensive process of research
instrumentation, the researchers started off with brainstorming and specifying with the
statement of purpose which was based on the curriculum shift happening in the
Philippine educational system. The existence of the 2 years allotment for the senior high
school including its new research curricular offerings ignited the researchers’ interest to
develop an instrument that seeks to determine their competence and appreciation
towards research after taking up such curricular offerings. After this step, the
researchers approached stakeholders to obtain feedback. Research directors,
instructors and potential respondents were likewise consulted and sought for
constructive feedback vis-à-vis the clarity of purpose.

After putting the stakeholders, experts and potential respondents’ feedback into
consideration, reviewing of tons to research literature and studies was done to gain
breadth and depth of knowledge of the research purpose. After which, planning for its
methodology including the type instrument best suits to the nature of the purpose was
the next step. Then the construction of the research items followed. The researchers

4
started by identifying the constructs and operationally defining them. Past of this phase
was the formulation of the table of specifications (TOS) as shown in the table 1. The
survey items were generated based from the identified and operationalized constructs.

Table 1. Table of Specifications

Constructs Operational Definition Sample Items
Research The ability to undergo the 1. Conceptualizing the problem
Competence dynamic of the research 2. Scanning available literature
process and the writing of 3. Consulting with adviser or other
the research technical expert about
paper.
Research This refers to the 1.Research is interesting.
Appreciation researcher’s ability to 2. Research is exciting.
recognize research as 3. Research is meaningful
useful, meaningful and
enjoyable.

To proceed, the survey items were reviewed by 4 content experts where each of
their suggestions was considered. Subsequently, these survey items were pre-tested
by 10 potential respondents. During the pretesting phase, focused group discussion
(FGD) was conducted to obtain feedback, clarifications, and even technical
suggestions. All these feedbacks were considered for revisions of the survey tool.
Overall, the survey questionnaire underwent 7 revisions, while the TOS had 3 revisions.
After all the revision, the final draft of the instrument was administered to the150
potential respondents for pilot-testing. The reliability and validity of the survey
instrument were computed via Statistical Package for Social Sciences (SPSS) trial
version before it was confirmed as ready to be administered.

Reliability and Validity Analyses

Colton and Covert’s (2007) discussed that in order of an instrument to obtain
reliable and valid information, evidences of validity and reliability must be established.
Validation in research refers to the compendium of the evaluative summary of
evidences (Sorono-Gagani & Bonotan, 2017). Research validity has many kinds
namely: face validity, content validity, criterion validity and construct validity.

Face validity is regulated by examining the appearance of the instruments

analysing whether the format and the content of the survey items indeed intend to
measure what is intend to measure. In order establish face validity evidences, the
researchers interviewed stakeholders, research experts, and potential respondents to
seek feedback. Suggestions and consideration were considered to revision.

To establish content validity, it is a must to seek help of the content experts to

provide suggestions and corrections to better design the survey questionnaire. The

5
experts scrutinized whether the items in the questionnaire are attuned and
representative of the operationally defined constructs. With this, the 2 researcher
directors and 2 research instructors were approached to validate the content validity of
the instrument.

Whereas, criterion validity is a validity test that requires the instrument designers
to compare the responses of the respondents to the responses to items in other similar
existing instruments. To establish criterion validity, the responses to other existing
instruments that measure the same variables were analyzed. The instrument designers
found high correlation in the data or the responses produced by the instruments.

Construct validity analysis is conducted to gather evidences showing strong

relationship among the items in the instrument. Such evidences could be produced
through item and factor analyses. To determine the underlying factors that contribute to
the relationship among the items, Exploratory Factor Analysis (EFA) was conducted.
The determination of items that are considered representative of a particular factor was
based on the Eigenvalues of over 1.00. Specifically, the Principal Component Analysis
(PCA) as extraction method was used. Table 3 shows these values which were
considered for the revision of the instrument.

Furthermore, research reliability refers to the internal consistency of the scores of

the instruments. In this instrument, reliability was determined using Cronbach’s alpha
which was calculated using SPSS. The researchers also used Cronbach’s alpha
coefficient value of at least 0.7 for item analysis. Reliability analysis measures are
shown in Table 2 in the results and discussion section of this paper.

Results and Discussion

Using the reliability statistics and item-total statistics for reliability analysis, the
research competence construct on its first round results of the pilot testing, result
revealed that 4 of the 44 items had coefficients of “Cronbach’s alpha if item deleted”
greater than the overall Cronbach’s alpha of .893. These figures helped the instrument
designers to delete the 4 items from the instrument making it a 40-item survey.
On the other hand, the results on the first round of pilot-testing of the research
appreciation construct revealed that 5 of the 17 items had coefficients of “Cronbach’s
alpha if item deleted” greater than the overall Cronbach’s alpha of .659. This notified the
instrument designers to delete the 5 items from the instrument leaving the item 12. After
revision, the survey was prepared for the next round of pilot testing.

6
Table 2. Cronbach’s Alpha Coefficients During Pilot Testing
Research Competence Construct Research Appreciation Construct
Pilot-Test Overall No. of No. of Pilot-Test Overall No. No. of
Periods Cronbach’s Items Items Periods Cronbach’s of Items
Alpha Deleted Alpha Items Deleted
Round 1 .935 44 4 Round 1 .659 17 5
Round 2 .970 40 0 Round 2 .906 12 0

The second round results of both constructs were likewise subjected to

exploratory factor analysis. As already mentioned, the determination of items that are
considered representative of a particular factor was based on the Eigenvalues of over
1.00. For the research competence construct, the factor analysis conducted after the
second round of pilot testing resulted to only 7 components. On the other hand, the
research appreciation construct resulted into 2 components as shown in Table 3.

Table 3. Initial Eigenvalues

Research Competence Construct Research Appreciation Construct
Component Total Cumulative % Component Total Cumulative %
1 18.701 46.753 1 6.318 52.646
2 2.249 52.376 2 1.872 68.242
3 1.894 57.111
4 1.533 60.943
5 1.257 64.085
6 1.099 66.832
7 1.034 69.416

Based on the same factor analysis of the results of the second round pilot
testing, the items that are representative of a particular factor or component were
identified. The instrument designer used the Varimax Rotation to identify these items,
when loadings less than 0.4 were excluded due to poor factor loadings. Table 4 and 5
shows the Rotated Component Matrix of the two constructs after applying the Varimax
Rotation method with Kaiser normalization, capturing a cumulative variance of 69% for
the research competence and 68% for the research appreciation of the variability of the
construct, students’ readiness for open and blended learning

Table 4. Rotated Component Matrix of the Research Appreciation Construct

Component
Items 1 2 3 4 5 6 7
Collating and organizing gathered... .722
Determining the research procedure .696
Analyzing the data using appropriate methods .646
Constructing research questionnaire .642
Specifying the data gathering Procedure .641
Planning on the data gathering process .639
Administering the questionnaire .634 .426

7
Determining the research locale .630 .446
Stating the research locale .572
Searching valid and reliable research instrument .527
Presenting the results and discussion .503
Making Conclusions .450
Generating conclusion .728
Formulating recommendation addressing the problem
.682
of the study
Integrating the literature to affirm or negate the results .617 .406
Writing the ethical consideration .561
Constructing the research abstract .558
Synthesizing the results of the study .551
Writing the rationale/ introduction of the study .487 .407 .438
Constructing research questionnaire .484
Writing with correct grammar .481
Specifying the sampling design .808
Specifying the research design .797
Estimating sample size .716
Specifying the statistical treatment of the data .409 .560
Creating the theoretical/ conceptual model .424 .541
Formulating the hypothesis of the Study .481
Utilizing online (e.g. Google Scholar) .811
Communicating with the research focus during the
.744
data collection phase
Observing APA .621
Formatting the research paper e.g. paging, spacing,
.612
bolding
Stating the research informants/ Respondents .581
Observing research ethics .479 .443
Scanning available literature .788
Consulting with adviser or other expert about the logic
of the problem, theoretical underpinning, and .666
research method
Conceptualizing the problem .445 .488
16. Presenting the gathered data .636
Analysing and interpreting the Findings .420 .580
Consulting the adviser for content validity of the
.576
instrument

Table 5. Rotated Component Matrix of the Research Appreciation Construct

Component
Items
1 2
Enhances my critical thinking skill .902
Uplifts my professional qualification .871
Has lots of benefits .869
Enables me to learn a lot of information .859

8
 Useful for my teaching career .849
Needed for my personal growth .790
.Improves my writing skill .783
Challenging .683
Fulfilling .637
Exciting .785
Fun .755
Interesting .414 .637

The results of both validity and reliability tests led the researchers to prepare the
final draft of the instrument for administration. Moreover, the results of both tests gave
confidence that the instrument will produce the same results over a period of time even
if administered on different occasions. Based on the same results, this instrument had
produced evidences that it would measure what it intends to measure as far as the open
and blended learning readiness of high school students is concerned.

Conclusion
The survey questionnaire on students’ research competence and appreciation
had passed through reliability and validity examination. For validity, it has gathered
evidences for face validity, content validity, criterion validity and construct validity.
Likewise, it has also established reliability. Therefore, it has proven that it is reliable and
valid and is now ready to be administered to the real respondents of the final study.

References
Borders, L. D., Wester, K. L., Granello, D. H., Chang, C. Y., Hays, D. G., Pepperell, J.,
& Spurgeon, S. L. (2012). Association for Counselor Education and Supervision
guidelines for research mentorship: Development and implementation. Counselor
Education and Supervision, 51(3), 162-175.
Colton, D., & Covert, R. W. (2007). Designing and constructing instruments for social
research and evaluation. John Wiley & Sons.
Ellis, T. J., & Levy, Y. (2008). Framework of problem-based research: A guide for novice
researchers on the development of a research-worthy problem. Informing
Science, 11.

Ellis, T. J., & Levy, Y. (2009). Towards a Guide for Novice Researchers on Research
Methodology: Review and Proposed Methods. Issues in Informing Science &
Information Technology, 6.

Ellis, T. J., & Levy, Y. (2010). A guide for novice researchers: Design and development

9
 research methods. In Proceedings of Informing Science & IT Education
Conference (InSITE) (Vol. 10, pp. 107-118).

Gómez, M., & Panaligan, C. (2013). Level of research competencies and satisfaction of
the faculty members from the college of criminology. ASian Academic Research
Journal of Social Science and Humanities, 1(14), 39-55.

Hollingsworth, M. A., & Fassinger, R. E. (2002). The role of faculty mentors in the
research training of counseling psychology doctoral students. Journal of
Counseling Psychology, 49(3), 324

Lambie, G. W., & Vaccaro, N. (2011). Doctoral counselor education students' levels of
research self‐efficacy, perceptions of the research training environment, and
interest in research. Counselor Education and Supervision, 50(4), 243-258.

Levy, Y., & Ellis, T. J. (2006). A systems approach to conduct an effective literature
review in support of information systems research. Informing Science, 9.

Magnuson, S., Norem, K., & Lonneman‐Doroff, T. (2009). The 2000 cohort of new
assistant professors of counselor education: Reflecting at the culmination of six
years. Counselor Education and Supervision, 49(1), 54-71.

Mallari, M. Q., & Santiago, M. M. (2013). The Research Competency and Interest of
Accountancy Faculty Among State Colleges and Universities in Region
III. Review of Integrative Business and Economics Research, 2(1), 51.

Meerah, T. S. M., Osman, K., Zakaria, E., Ikhsan, Z. H., Krish, P., Lian, D. K. C., &
Mahmod, D. (2012). Measuring graduate students research skills. Procedia-
Social and Behavioral Sciences, 60, 626-629.

Papanastasiou, E. C. (2005). Factor structure of the attitudes toward research

scale. Statistics Education Research Journal, 4(1), 16-26.

Şahan, H. H., & Tarhan, R. (2015). Scientific research competencies of prospective

teachers and their attitudes toward scientific research. International Journal of
Psychology and Educational Studies, 2(3), 20-31.

Şorono-Gagani, F. & Bonotan, A.M.(2017). Developing and Validating an instrument to

Evaluate a Mathlete Training Program. Asia Pacific Journal of Education, Arts
and Science. 4 (1).1-9.

Swank, J. M., & Lambie, G. W. (2016). Development of the research competencies

scale. Measurement and Evaluation in Counseling and Development, 49(2), 91-
108.

10