The Adequacy, Appropriateness and Correctness of Statistical Analyses

In Graduate Students Theses/Dissertations in

Higher Education Institutions
In Bicol Region, Philippines

By

EDDIE S. SEE,
College of Business, Economics and Management,
Bicol University, Daraga, Albay ess4560@yahoo.com

MARY ANN M. SEE

Associate Researcher, General Administration,
Bicol University, Legazpi City meannms@yahoo.com

Abstract

This study sought to evaluate the adequacy, appropriateness, and correctness of

statistical analyses in masters theses and doctoral dissertations in higher education
institutions in the Bicol Region, Philippines . Applying content analysis upon existing 78
manuscripts (55 masters theses and 23 doctoral dissertations) from eight
colleges/universities in three Bicol provinces, probing the survey data with frequency
count, percentage and ranking and applying insight, empathy, intuition, and the
elements of art and reasoning, imagination, and discernment; and making
comparisons with discourses of experts, these researchers considered that they are
not-so-adequate, not-so-appropriate, and partially correct in terms of statistical analyses
primarily because of deficient methodology and research design.

Introduction

Rationale

Research plays a very vital role in global development. It has caused and still
effects the entry of technology that facilitates mans activities, made medical
breakthroughs that save and prolong the lives of billions of people, revolutionized
farming that tremendously increase the food yields of lands and helped build facilities
that significantly lengthen the storage life of edibles. On the other hand, research also
challenged long held traditions, defied norms, proved popular predictions wrong, and
disputed previously accepted theories. Amid stiff opposition and even threats of
banishments, research braved all these and still prevailed because of its objective,
correct and scientifically respected approaches.

The use of appropriate statistical tools to analyze data is one way of coming up
with objective, correct, and scientifically respected research results and
recommendations. While the use of statistical methods is important, it is likewise skills-
based, and needs to be properly applied. Statistical application sees to it that data are
treated appropriately. The academe plays a major responsibility in ensuring this.

The application of statistical tools to quantitative measures of problem variables

is a major component of any thesis or dissertation. Statistics involves the collection,
organization and analysis of data. Before any statistical analysis can be made, data
must be appropriately collected and systematically organized. Even in the advent of
statistical programs, the scientific collection and organization of data are indispensable
in statistical analysis.

It must be noted that as research works, theses and dissertations are scientific
investigations whose methodology must be so designed and applied to warrant further
investigation or tests by other researchers. The absence of required data and
computations or cited statistical programs needed to show proof or basis to presented
statistics and critical values in a thesis or dissertation could cast doubts on the credibility
of the findings being presented in the report. Research designs, that include sampling
design and analytical tools, are necessary to guide the investigator as to what type of
statistical method to apply.

Objectives

This research proposed to find answers to the following questions:

1. What descriptive and inferential statistical analyses were employed in

masters theses and doctoral dissertations in higher education institutions in
the Bicol Region from year 2000 to year 2007?
2. How adequate are these statistical analyses?
3. How appropriate are the statistical tools used in these theses and
dissertations, vis--vis:
a. Completeness of the sampling procedure
b. Consistency between the sample type and the statistical method
c. Consistency between the problem and the sampling
d. Consistency between the problem and the statistical method
4. To what extent are the statistical tools correctly applied and the results
interpreted?
5. What training curricula may be proposed to fill the shortcomings identified by
the study, if any?

Review of Related Literature

As an applied research, this study is neither founded on nor does it test any
existing theory. It is however, open to the likelihood of being able to provide grounds for
the development of theory on the proposition, conduct and reporting of graduate student
researches vis--vis adequacy, appropriateness, and correctness of statistical
analyses..
 Studies assessing the methodological flaws in published researches have been
conducted. Onwuegbzie and Leech (2005) were able to identify six major flaws that
include blunders in sampling, in book used in colleges. The Paul Cameron Groups
Surveys that were published have been criticized for erroneous, misrepresentative
sampling, such that their results cannot be considered a valid description even of the
specific group of individuals who returned the survey questionnaire (___, 2005). Of the
six peer-reviewed papers studied by Stephens (2004), three were found to have serious
flaws in methodology related to data collection and interpretation.

Appropriate statistical applications are important to research. This matter has

been classically illustrated in the case of the presidential elections in the United States
of America in 1936. The Literary Digest, with non-random selected respondents of
2,376,523 wrongly predicted that Landon would win over Roosevelt, while George
Gallup, with a low but randomly selected voters of 3,000, correctly foretold that
Rossevelt would win (Sprinthall, 1997).

The significant existence of blunders in statistical usage has not been left
unnoticed by the research community. This fact is one of the primary reasons why
Journals are supposed to be refereed nowadays. However, student researchers-
undergraduate, and graduate levels, do not have a refereeing system after they have
been submitted as final manuscripts. The evaluation of these documents is assigned to
a panel of experts. Despite review by specialists, however, many of these theses and
dissertations remain wanting, especially in the application of statistics. This has been
noted by this researcher in a number of papers he personally browsed through.

While there are critical analyses of published researches, refereed and not, there
seems to be an absence of critical research on unpublished theses and dissertations.
This dearth is the gap that this investigation seeks to bridge.

Materials and Methods

This investigation employed the cross-sectional, quantitative-qualitative, ex-post

facto design. Its population encompassed the thesis/dissertation final manuscripts
submitted by graduate students from year 2000 to year 2007in the higher education
institutions in the Bicol Region as a requirement for graduation, and available in their
respective libraries. The size of the population was deemed unknown by the researcher
for practical reasons, although, theoretically, the size was in fact finite.

The sample size of 78 manuscripts (23 dissertations, 55 masters theses) was

purposive quota. This number was primarily based on the budget available for the
conduct of the study. The 78 individual manuscripts were selected via convenience
sampling from eight (8) higher education institutions coming from three provinces of
Bicol Region, Philippines. Materials available in the libraries visited during the data-
collection period were picked.
 This research made use of documentary survey (existing data search) and
evaluation (critical analysis). Research reports, as Jupp (1996) puts it, are themselves
documents and as such are not immune from critical analysis This method of analysis
embraced reading and cogitating upon the research problems, the research design,
the sampling design, the statistical tools, the data presented, and the computations
made in the thesis and dissertation manuscripts, peeping closely at the way in which
they were used vis--vis and guided by the authoritative norms on the subject,
especially the basic statistical concepts. And while the analysis is at best anchored on
authoritative norms of statistics usage, it still depended on the limited expertise of the
analyst, and therefore invites re-analysis or further scrutiny from his peers. This mode of
study was made through the use of a documentary survey guide and a content analysis
guide.

To probe the documentary survey data, frequency count, percentage and ranking
were applied. The analysis, adapting the definitions of Lewis (1953) and Arce (2001),
also entailed the use of insight, empathy, intuition, and the elements of art and
reasoning, imagination, and discernment. It also made comparisons with discourses of
experts as Wolcott (1994) wrote, qualitative data analysis draws connection with
external authority.

Results and Discussions

Use of measures of spread

Data show much greater use of measures of centrality (arithmetic mean,

weighted mean, mode through frequency count) over measures of spread (standard
deviation) as descriptive tools. Only one out of 32 included a measure of dispersion-
standard deviation. This seeming bias in favor of normative data analysis dates back
to the 1800s when Sir Francis Galton (1889) as cited by Freedman, et.al (1991)
lamented

It is difficult to understand why statisticians limit their enquiries to

AveragesTheir souls seem as dull to the charm of Variety as that of the nature
of our flat English countries.

The importance of describing data in terms of how they part from the mean value
is never less than that describing them in terms of averages. Variability shows the
inconsistencies in the values (Vincent, 1995). Information about variability is often at
least as important as averages. A manufacturer is always interested in the consistency
of the quality of his product as well as the level of quality (Minium, 1978). Glass &
Stanley (1970) stress the importance of measuring dispersion by averring

The whole scientific enterprise is concerned with notions of variability.

When much unexplained variability exists, predictions can never be made. But
 when explanations can be fashioned for why people or things differ, portions of
variability can be removed.

One vital import of the variance (or standard deviation) is its worth in the
computation of sample size in future studies involving the same population. It is to be
noted that sampling formulae require an input of population variance that can be based
on the sample variance of previous researches. As Davis, Utts and Simons (2002)
indicated in their sampling formulae for ratio variables, sample size computation
requires the input of population variance, and for categorical variables, the population
proportion.

Inferential statistical analyses

The table demonstrates chi square test of relationship as the most frequent
inferential statistical tool used, followed by 2-way ANOVA, then by T-test of correlation,
and by one-way ANOVA. Apparently, the inferential statistical analyses employed in
masters theses and doctoral dissertations in the graduate schools are limited to these
methods.

Table 1. Frequency distribution by statistical analyses

Statistical analyses applied Frequency of Rank
theses/dissertation
s (multiple
response)
Descriptive
Frequency count 32 1.5
Percentage 32 1.5
Arithmetic mean 26 3
Weighted mean 22 4
Ranking 1 5.5
Standard deviation 1 5.5

Inferential
Chi-square test of association 12 1
Two-way ANOVA 9 2
T-test of correlation 8 3
One-way ANOVA 6 4
T-test of dependent samples 5 5
T-test of independent samples 3 6
Kendal-Tau 2 8.5
Scheffes test 2 8.5
Median test 2 8.5
Z test 2 8.5
Friedman test 1 11.5
Tukeys test 1 11.5
 Adequacy, Appropriateness and Correctness of Statistical Analyses

Adequacy

Adequacy refers to the sufficiency of organized data for another researcher or

statistician to verify the statistical computation.

In general, the statistical analyses in the graduate student researches are not so
adequate, meaning that while data presented in the discussions are organized in such
a manner where a statistician can observe the value trend between variables being
compared without necessarily resorting to computations, they are not sufficient for one
to re-compute the statistical method applied. Only 55 (of the 78 papers) appended the
profiled participants data in the manuscript.

The absence of valuable data that can be used for re-computation in the
manuscripts is not an inadequacy on the part of the researcher. Generally, graduate
schools in the province do not require their inclusion in the manuscripts. Nevertheless,
the merit of adding them in cannot be underestimated. The usefulness of such data is
not just on the research conducted but for future undertakings as well, especially for
carrying out secondary analysis.
.
Appropriateness

Appropriateness refers to the fitness of the statistical tool/s employed

In general the statistical analyses in the graduate student researches are not-so-
appropriate meaning that the tools used do not jibe with some or all of the following:
the research problem, the sampling procedure, or the sample type, The inappropriate
portions of the analysis are manifested in the following:

Completeness of the sampling procedure

Completeness of the sampling procedure refers to the presence of the basic

components in sampling, namely population identity and size (for finite participants),
sample size computation stating the confidence level and margin of error and sample
selection for probability samples, or sample size determination and selection for non-
probability samples.

Sampling procedure encompasses identifying the population including its scale

and size; pinpointing the general sampling type, and the probability sampling nature or
the non-probability sampling category; ascertaining the sample size computation
formula and classifying the mode of sample element selection.

Fourteen (17.95%) of the papers had complete sampling procedure. The

remaining 64 (82.05%) all had only partial sampling process. This deficiency in the
sampling includes not mentioning the population or the general sample type or the
probability sample class or the sample selection. Some even claimed of probability
sampling yet also named convenience sampling.

Frequently observed was that many papers that executed inferential analysis did
not include in their methodology either the information on how the sample elements
were selected or how the size was selected or both The missing portions make
sampling entirely vague if not totally incomprehensible.

The methodology section describes a studys measures, sample, and research

design where the discussion of the sample identifies the target population, sampling
frame and sampling design (OSullivan & Rassel, 1995). It contains the sampling
design, the research design, the data collection, the data analysis, and the limitations
(Cooper & Schindler, 1998). A subsection in the methodology on participants or
respondents where their nature and other relevant characteristics should be described
is always necessary (Brown, et. al., 1999).

Methodologies that are wanting in the basic facets of sampling are manifest in
the theses and dissertations surveyed.

Consistency between the sample type and the statistical method

Consistency between the sample type and the statistical method refers to a
probability sample applied with inferential statistics and non-probability sample operated
with descriptive statistics,
Thirty (38.46%) of the manuscripts have consistent sample type and statistical
method. The other 48 (61.54%) had incompatible sample and statistical tool, most of
them non-probability samples employed with inferential analysis and probability
samples where no inference was made- especially for one-variable (non-comparative)
analysis.

The application of inferential analysis to non-probability samples, that is,

employing tests of significance such as t-test of correlation, t-test of two samples, and
analysis of variance to samples that are either accidental or purposive quota is a plain
blunder in hypothesis testing.

Probability sampling is the only one general approach that allows the researcher
to use the principles of statistical inference to generalize from the sample to the
population (Frankfort- Nachmias & Leon-Guerrero 2002), its characteristic is
fundamental to the study of inferential statistics (Davis, Utts & Simon, 2002). It is the
sampling technique that uses the probability theory to calculate the likelihood of
selecting a particular sample and allows the drawing of conclusions about the
population from the sample.(Pelosi, Sandifer, & Sekaram, 2001) and it has the
advantage of projecting the sample survey results to the population (McDaniel & Gates,
2002). Inferential statistical analyses are based on the assumption that the samples
being analyzed are probability samples (Burns & Grove 1997).
 The results are a manifestation of the indiscriminate significance test mindset
that developed among researchers whose studies involve the comparison of values
between two or more groups. This occurs perhaps because of many peoples habit of
imitating statistical tools in previous reports without analyzing carefully the incidents
covered in such papers. This also indicates that many researchers do not fully
understand the meaning of statistical inferential analysis.

Consistency between the problem and the sampling

Consistency between the problem and the sampling refers to a descriptive

problem having a non-probability sample and an inferential problem employed with a
probability sample.

Thirty one (39.74%) of the researches had consistent problems and sampling.
Forty-seven (60.26%) of them had problems and sampling which are not in agreement.
Mostly, this includes problems of Is there a significant relationship? or Is there a
significant difference? that made use of purposive quota and convenience sampling.

Consistency between the problem and the statistical method

Consistency between the problem and the statistical method refers to a

descriptive problem utilizing descriptive statistical tool and an inferential problem
operating inferential statistical tool.

Sixty-one (78.21%) of the materials had consistent problems and statistical tools.
Seventeen (21. 79%) had conflicting problems and methods.

Weinbach and Grinnel (1995) suggested an explanation for this situation.

Why are inappropriate tests chosen when appropriate ones are

available for almost all types of data analysis? One reason may be that
some people have a tendency to see the solution to any problem as
requiring what they know and/or do best that is, with which they are
most comfortable and familiar.

In totality, this inappropriateness may partly be due to or is already a

manifestation of an unclear, deficient methodology, particularly the research design.
Burns and Grove (1997) noted that many published studies do not identify the design
used in the study. Research design and statistical analysis are so intertwined that it is
difficult to discuss one without referring to the other (Vincent, 1995).

The research design constitutes the blueprint for the collection, measurement,
and analysis of Data (Cooper & Schindler, 1998). As part of the Methodology, the design
indicates the basic structure of the study including the nature of the hypothesis, the
variables involved, and the constraints of the real world (Gay, 1992). It includes a
description of the participants, the size of the sample and a discussion of the sources of
invalidity associated with the design (Gay and Airasian, 2000).

Davis and Cosenza (1988) aver that the design process is a series of decisions
concerning what concepts will be studied, how these concepts will be measured, what
approach will be used to study the problem, who will be studied, how the data will be
collected and analyzed, and, ultimately, how will the information that was collected be
presented to solve the problems. The research design has two basic purposes- to
provide answers to research questions and to control variance- and the research
question should be a common unifying element in the design of research (Kerlinger,
1973).

Users of research must be concerned with the issue of research design because
an inadequate design will yield information that is incapable of answering the research
question under study (Davis, Utts & Simon, 2002). In assessing the quality of content
and the merit of a research paper, Bartol (1981) as cited by the American Psychological
Association (1994) lists, among others the clear relatedness of the outcome measures
to the variables with which the investigation is concerned

It is generally observed that a research design must be able to capture the

studys population (identity and size if finite), the kind of sample depending on the
nature of the research problem (probability, non-probability), the determination of the
sample size depending on the kind of sample (computed based on margin of error and
confidence level and population size if the population is known, purposive quota), and
the type of sample element selection (random sampling, snowball).

As seen and analyzed, majority of the manuscripts evidently were insufficient

either in the methodology as a whole or in the research design in particular.

The other possible cause of this situation is a lack of lucid knowledge about the
importance of the relationship between the kind of sample and the applicable statistical
method.

Correctness

Correctness refers to the accuracy of computation and correctness of the

interpretation.

Generally, the analyses are partially correct that is, the computed and tabulated
values are accurate vis--vis the accepted formulae, in so far as manuscripts that
appended/contained re-computable data are concerned, but the interpretation is
incorrect vis--vis the statistical hypothesis.
 The incorrect interpretation is understandably brought about by the tools
inappropriateness, or the use of inferential analysis for non-probability samples. This is
also an implied manifestation of the researchers insufficient understanding of the
commonly encountered tests of significant difference/relationship. Hamilton (1996)
presented a relatively easy-to-understand definition for this expression.

Statistically significant means the sample provides evidence against the

null hypothesis (no population relation/difference). We therefore conclude in favor
of our alternative hypothesis (some population relation/difference). This some
relation/difference conclusion does not necessarily mean that the relation/
difference is strong/big, or has practical importance. In large samples, even a
weak relation/small difference possessing no practical importance could be
statistically significant. Not statistically significant means the sample lacks
convincing evidence against the null hypothesis. This conclusion does not
disprove the alternative hypothesis. However. We find insufficient support for it in
our sample. In small samples, even important strong/big relations/differences
may not be statistically significant.

Conclusions and Recommendations

Conclusions

In so far as the manuscripts covered are concerned, the conclusions of this

research can be summarized into the following:

Measures of spread (such as standard deviation and variance) are not as much
used as measures of centrality (such as arithmetic mean and mode) for descriptive
studies by in graduate students in the papers assessed. Chi square test of association
and two-way analysis of variance are the most used inferential statistical tools in the
theses and dissertations studied.

The manuscripts are not-so-adequate, not-so-appropriate, and partially correct in

terms of statistical analyses. They are not so adequate primarily because of the non-
availability of the data to warrant re-computation by a statistics-knowledgeable reader;
they are not-so-appropriate because of the presence of inconsistencies between the
research problems, the sampling and the statistical method; and they are partially
correct because of incorrect interpretations brought about by the use of inferential
analysis to non-probability samples.

While the sampling done in this undertaking is non-probability, the number (and
the proportion vis--vis the total of 78) of the manuscripts in situations of inadequacy,
inappropriateness and incorrectness (please see table 2) can be said to be substantial
enough in magnitude to warrant a re-visit to the proposition and conduct of masters
theses and doctoral dissertations in higher education institutions in Albay.
Table 2. Frequency and percent distribution by manuscript situation
Manuscript situation Frequenc Total, N Percent
y in such
situation,
n
Lacking in data to warrant re 23 78 29.49
computation (not-so-
adequate)
With incomplete sampling 64 78 82.05
With inconsistent sample type 48 78 61.54
and statistical method
With inconsistent problem and 17 78 21.79
statistical method
With incorrect interpretation of 48 78 61.54
statistical results

Recommendations

Based on the foregoing conclusions, the following recommendations are hereby

forwarded:

1. The development and implementation of training curricula that could address

the issues on the identified adequacy, appropriateness and correctness of
statistical analyses, namely hands-on trainings by would-be professors in
research, research advisers and evaluators, and proponents of
theses/dissertation proposals on:
a. Research designs, giving emphasis on practical illustrations
connecting the research problem, the sampling design, and the
statistical analysis tools.
b. Statistical tool-friendly data coding and encoding.
c. Sampling designs

2. The review and possible enrichment of the contents of masters theses and
doctoral dissertations focusing on the issues emerging from these
conclusions, particularly the methodology- research design components,
sampling design components, and statistical methods.

3. The conduct of periodic reviews on the subjects with would-be researchers,

advisers and evaluators.

4. The assignment of Advisers and Evaluators to student research defense who

have the competence to address the issue.
References

American Psychological Association. 1994. Publication Manual. Tenth Printing

August 1998. American Psychological Association, 750 First Street, NE Washington DC
20002.

Arce, Wilfredo. 2001. Systematic Qualitative Data Research An Introduction for

Filipino Practitioners. Second Edition. Office of Research and Publications. Ateneo de
Manila University.

Brown, Kathleen, P. Cozby, D. Kee & P. Worden. 1999. Research Methods in

Human Development. Second Edition. Mayfield Publishing Compnay. London.

Burns, Nancy & S.K. Grove. 1997. The Practice of Nursing Research Conduct,
Critique and Utilization. 3rd ed. W.B. Sande4rs Co. The Curtis Center Independence
Square West Philadelphia, Pennsylvania 19106.

Cooper, Donald & P. Schindler. 1998. Business Research Methods. Irwin-

McGraw-Hill. New York.

Davis, Duane & R. Cosenza. 1988. Business Research for Decision Making.
PWS-Kent Publishng Co. Boston.

Davis, Duane, J. Utts & M. Simon, 2002. Statistics and Research Methods for
Managerial Decision. Second Edition. Brrok-Cole. 511 Forest Lodge Road. Pacific
Grove, CA 93950, USA.

Frankfort- Nachmias & Leon-Guerrero 2002. Social Statistics in a Diverse

Society.

Freedman, David, R. Pisari, R. Purves, & A. Adhikari. 1991. Statistics. Second

Edition. W.W. Norton and Company, N.Y., U.S.A.

Gay, L.R. 1992. Educational Research Competencies for Analysis and

Application. Fourth Edition. Merril Publishing Company. New Jersey.

Gay, Lorraine and P. Airasian. 2000. Educational Research Competencies for

Analysis and Application. Sixth Edition. Merril Publishing Company. New Jersey.

Glass, Gene V. & J. Stanley. 1970. Statistical Methods in Education and

Psychology. Prentice-Hall, Inc., Inglewood Cliffs, New Jersey.
 Having methodological flaws cannot be reliably used. 2005. News-Medical. Net.
Online Available http://www.news.medical.net/ April 3, 2009
Kerlinger, F. 1973. Foundations of Behavioral Research. 2nd ed. Holt, Rinehart &
Winston, NY, USA.

Lewis, Oscar. 1953. Controls and Experiments in Field Work in Anthropology

Today. A.L. Kroeber, ed. University of Chicago Press, Chicago, USA.

Mc Daniel, Carl & R. Gates. 2002. Marketing Research: The Impact of the
Internet. Southwestern Thomson Learning 5101 Madison Road, Cincinnati, Ohio 45227.
http://www.swcollege.wm

Minium, Edward W. 1978. Statistical Reasoning in Psychology and Education. 2 nd

Edition. John Wiley & Sons, NY.

Pelosi, Marilyn, T.M. Sandifer, & U. Sekaram. 2001. Research and Evaluation for
Business. John Wiley and Sons, Inc. 605 3rd Vaenue, NY, NY.

Onwuegbuzie, Anthony & N. Leech. 2005. Current Issues in Education. Volume

8, 2005. Mary Lou Fulton College of Education. Arizona State University.Online.
Available http://www.cie.asu.edu April 3, 2009

OSullivan, Elizabethann & G. Rassel. 1995. Research Methods for Public

Administrators. Second Edition. Longman Publisgers USA. NY.

Sapsford, Roger & Victor Jupp 1996. Data Collection and Analysis. Sage
Publications. London.

Sprinthall, Richard. 1997. Basic Statistical Analysis. Fifth Edition. Allyn and
Bacon. Boston.

Stephens, Jim. 2004. Evaluating Research. IFF Academy Feldenkrais

Research Journal. 2004. Online. Available http://iff.researchjournal.org/ April 3, 2009.

Vincent, William L. 1995. Statistics in Kinesiology. Humankinetics, P.O. Box

5076, Champaign, IL.

Weinbach, Robert W. & R. M. Grinnel, Jr. 1995. Statistics for Social Workers.
Third Edition. Longman Publishers USA, Longman, 10 Bank Street, White Plains, NY,
10606.

Wolcott, H.F. 1994. Transforming qualitative data: Description, analysis, and

interpretation. Sage Publications, Thousand Oaks, CA.