The challenge of conducting a quantitative business research:

Analysis of issues with survey design, sampling, validity, and reliability

Dr. Komlan Joel Adzeh

Ph.D. in Organization and Management
Alumnus,
Capella University School of Business and Technology
Capella Tower, 225 South Sixth Street, Ninth Floor, Minneapolis, MN 55402,
Email: kadzeh@capellauniversitu.edu

Abstract:
The aim of this literature research article was to provide insight into the challenges that
quantitative researchers face and to analyze the specific issues with surveys. Several studies have
been synthesized and analyzed. Building upon the current knowledge of the field, this article
highlighted some specific areas of concern, including sampling, validity, and reliability. While
survey research is a method of choice among quantitative scholars and practitioners, the rigor
with which it is conducted within the business community remains a topic of interest. This article
indicates a direction to future research.
Keywords: Business Research, Quantitative Methodology, Survey Design, Sampling,
Validity, Reliability
1. Introduction
Businesses across the world collect information to gain deep insight about market trends,
competitors, consumer behaviors, potential opportunity for growth, and any kind of threats that
could inhibit their performance. Therefore, survey research has become a growing field with
20,000 job openings in the United States alone and an estimated growth rate of 24% over the
next decade, according to the Bureau of Labor Statistics, U.S. Department of Labor (2012).
As the demand for business research continues to grow, surveys have been extremely
popular among quantitative researchers and the design of choice to investigate several issues,
particularly in marketing and customer behavior. Besides, a survey design involves collecting
data from a representative sample with the intent to generalize the findings to a larger population
(Barlett, Kotrlik, & Higgins, 2001). In addition, several researchers argued that survey design is
both economical and capable of yielding a quick turnaround in the data collection process
(Couper & Miller 2008; Frippiat & Marquis, 2010). However, this process is not always
conducted with the same rigor in business as compared with the academia.

Although surveys play a key role in business quantitative research, they are conducted in
the real-world environments, which cannot be controlled or manipulated. Further, the decline in
survey response over the last decades represents the greatest difficulty researchers have ever
faced (Schmeets, 2010). This paper discusses the challenges related to the philosophical
foundation of survey research and analyzes issues involving sampling, validity, and reliability.
2. Challenges Related to the Philosophical Foundation of Survey Research
Survey research is grounded in the positivist paradigm according to which reality is
objective, unbiased, and completely independent of both the researcher and the subject
(Bielefeld, 2006; Johnson & Onwuegbuzie, 2004). According to Firestones (1987) seminal
work, the premise of this world view is the acceptance that social facts are objectively
measurable, and they are free from the researcher. Carr (1994) argued that this detachment
ensures the neutrality of the study and prevents biases.
Although this world view explains the reason why findings were historically attributed to
quantitative research, it was also criticized. First, Walsham (2006) contended that there is no
such thing as objective fact that researchers can learn. Instead, reality is socially constructed
through the meanings, interpretations, and interactions of individuals with events. Second, Guba
and Lincoln (1994) argued that the positivist paradigm removes meanings and interpretations
from data in order to quantify phenomena. Moreover, Clark (1998) claimed that different
philosophical assumptions, and not just only one, have shaped the quantitative research.
Therefore, he argued that it would not be appropriate to consider the quantitative research as
rooted in a single paradigm.
Not too long ago, Amaratunga, Baldry, Sarshar, and Newton (2002) questioned the
appropriateness of quantitative survey method and argued that it aims at taking a snapshot look
at situations, which can be affected by temporal changes. Probably, the fiercest criticism was that
the quantitative method neither recognizes the importance of resource constraints nor the
variability in human behaviors (Eldabi, Irani, Paul, & Love, 2002). Similarly, Johnson and
Onwuegbuzie (2004) agreed that the knowledge that quantitative analysis produces may be too
abstract and may not reflect the specificity of the contexts, situations, or understandings of local
constituencies. Because of the ontological dichotomy in conceptualizing the quantitative research
method, interpretivism and post positivism have emerged as alternative world views of
conducting social research.
Furthermore, Harrison, Hill, and Leitch (2010) contended that interpretive researchers
aim to understand true meanings of the social world by embracing its complexity. Whereas, post
positivists considered that there is only a probable reality, and no one can certainly comprehend
the nature of the truth (Ponterotto, 2005). Both world views have changed the landscape of social
research and have an implication for the business management. Recent studies suggested
blending different paradigms altogether into one study (Onwuegbuzie, Turner, & Johnson, 2007;
Terrell, 2012). While much progress has been achieved in studying social phenomena,
quantitative business scholars continue to be challenged with issues involving sampling, validity,
and reliability.

3. Challenges Related to Sampling

Survey researchers expect to obtain a minimum sample size to produce results that are
statistically reliable and generalizable as an inadequate sample can undermine the accuracy of
the findings (Barlett, Kotrlik, & Higgins, 2001). However, several challenges present a threat to
the precision of a survey (Grove, 2006). Chief among them is the non-response. Even though, a
stream of researchers contended that a change in non-response rate does not necessary change
survey estimates, it is generally recommended reducing non response rates as much as possible
(Curtin, Presser, & Singer 2000; Keeter, Miller, Kohut, Groves, & Presser, 2000).
Methods used to reduce non responses include advance letter (Hox, 2007), incentives, or
longer periods of data collection (Grove, 2006). While these techniques are generally successful
in reducing non response threats, they are costly to implement. Additionally, the limitation of
resources obliges survey researchers to turn to nonrandom sampling instead of devoting their
energy to try to obtain appropriate sample sizes or responses. As Carr (1994) explained, random
sampling is time-consuming and very often, researchers turn to opportunistic samples, which
they can obtain easily. The downside to this approach is that, it compromises the generalization
from the findings (Onwuegbuzie & Johnson, 2006).
4. Challenges Related to Validity
Quantitative survey researchers share the excitement that the conclusions they would
draw from analyzing data would be credible, objective, and reliable. At least, that is what they
expect. However, a variety of threats can occur during any stage within the research process,
which can compromise the validity to the study. Onwuegbuzie and Johnson (2006) compiled
these threats into three categories relative to the content, criterion, and construct validity of the
survey instrument.
4.1. Content Validity
Content validity refers to the degree to which survey instrument items are relevant and
representative of the construct they intended to measure (Rossiter, 2008). It includes face
validity, item validity, and sampling validity (Onwuegbuzie & Johnson, 2006). Face validity, in
particular, describes how well a survey instrument seems to measure what it is designed to
measure. Further, Collins (2003) explained that face validity referred to an informal look to the
instrument in order to ensure that it is appropriately designed to obtain the right information from
the population of interest. Therefore, survey researchers use field pretest to inquire feedback on
areas of weakness that may need improvement. Although suggestions from field pretest may
contribute to improve the quality of the questionnaire, Betts (2011) argued that researchers
should not solely rely on them because further refinement may be necessary before the
deployment of the instrument.
4.2. Criterion Validity
Criterion validity requires thoroughly testing the instrument by comparing the values
inferred from the test to the criterion values actually observed or by correlating those values with

other measures of similar behavior (Rejas, Monfort, Campillo, Ruiz, Pardo, & Soto, 2009).
Specifically, criterion validity includes concurrent validity and predictive validity (Onwuegbuzie
& Johnson, 2006). Further, survey researchers seek expert opinions on the appearance,
relevance, and the representativeness of all components of the survey instrument in order to
ascertain that the questionnaire measures the right construct (Turocy, 2002). This method
requires asking the exact questions, identifying the true criterion, having appropriate statistical
expertise, and being able to engage skilled reviewers. This process can be a challenge
sometimes. One notorious example was about Coca-Cola Company, which, in 1985, neglected to
ask consumers, whether they preferred the brand-new Coke flavor to the old one, which could
have been a perfect criteria for benchmarking the preferences before introducing the New
Coke (Shuttleworth, 2009). The consequence was what News and Media reported at the time as
one of the biggest marketing fiascoes in history simply because Coca-Cola Company neglected
to ask the central question of Coke users, Do you want a new Coke? (Ross, 2005).
4.3. Construct Validity
Construct validity is the extent to which survey instrument is free from measurement
errors (O'Leary-Kelly & Vokurka, 1998), and it addresses the representativeness of the scale, the
appropriateness of the items, and the carefulness of their articulation (Burton & Mazerolle, 2011;
Parasuraman, Zeithaml, & Berry, 1988). In addition, Cronbach and Meehl (1955) asserted in
their seminal work that, construct validation is needed whenever a researcher considers that the
proposed survey instrument represents a specific construct and to which particular meanings are
attached. In this case, it is another challenge to ensure that potential survey participants capture
the exact meaning that the researcher attaches to the construct. Therefore, researchers may need
to conduct field pretest several times to improve the construct validation of the survey.
4.4. External and Internal Validity
According to Ihantola and Kihn (2011), external validity refers to the generalization from
the study to other samples, other times, and beyond the initial setting. More importantly, they
argued that external validity determines whether researchers can draw general conclusions based
up on the model they proposed and the data they collected. Further, Ihantola and Kihn (2011)
explained that three typical issues might jeopardize the external validity of a quantitative
investigation: (a) inaccessibility to the population or having a sample that is not representative;
(b) time validity or whether conclusions can be generalized to other periods of time; (c) and
environmental validity or whether conclusions can still hold true across settings.
While Ihantola and Kihn (2011) defined internal validity as the measurement of
variations between dependent and independent variables, which are not due to other confounding
factors, Onwuegbuzie and Johnson (2006) emphasized that even skillful researchers can still be
challenged by numerous threats that occur at various stages of a research process. Building on
previous works, Onwuegbuzie and Johnson (2006) identified several threats to both internal and
external validity at the stage of research design and data collection. Issues such as Type I and
Type II Errors, violated assumptions, misspecification error, multicollinearity, distorted graphics,
confirmation bias, and causal error were identified as serious threats to both external and internal
validity of a survey.

Therefore, it becomes extremely important for survey researchers to minimize the effect of these
threats to the quality of their studies and specify the extent to which their estimates are reliable.
5. Challenges Related to Reliability
Thompson and Vacha-Haase (2011) argued that reliability helps to determine whether the
exact study can be replicated under the same condition, and the scale can produce consistent
results of measurements. In his seminal work, Churchill (1979) recommended the use of
Cronbachs alpha coefficient as the first measure researchers should calculate in order to assess
the quality of their data. He assumed that if all the items in a particular measure are measuring
the domain of a single construct, there should be a high inter correlation between the responses.
In other words, survey respondents will tend to score the same way across a sample of multiple
items due to the internal consistency of these items. Therefore, a low Cronbach alpha coefficient
indicates that the sample of items does not perform well in capturing the construct that the
instrument intends to measure. Whereas, a high alpha suggests that the items are the true
reflection of the construct, and they are relevant to assess the internal consistency of the
construct. Generally, survey researchers consider a minimum of .70 for alpha value to be
acceptable in reference to Nunnally (1978) study.
Despite the acceptance of Cronbachs alpha as the estimate of reliability and
measurement of the internal consistency of survey items (Cortina, 1993), researchers still face
challenges in reporting, interpreting, and using reliability coefficients because of the
contradictory advice commonly found in psychometric journals (Sass, Mifsud, Helms, & Henze,
2006). In a recent study, Thompson and Vacha-Haase (2011) found that 54.6% of authors did not
mention reliability in their reports. Only 15.7% did but simply reported previous scores as if they
were applied to their own data.
The issues surrounding the report, interpretation, and the utilization of coefficient alpha
raised some concerns. Sijtsma (2009) criticized the coefficient alpha as a true measure of the
reliability. Further, Green and Yang (2009, 2011) utterly discouraged its use altogether. The
former contended that alpha, by itself, is not a measure of internal consistency because it
depends on several other statistics, and it is quite difficult to test the same individual repeatedly.
In the same line of thought, Emons, Sijtsma, and Meijer (2007) suggested increasing the number
of items for the survey instrument in a way to gather as much information as possible about the
participants (Sijtsma, 2009). Although this may be practical, evidence also showed that long
questionnaires could encourage non responses in survey research (Bean & Roszkowski, 1995;
BeVier & Roth, 1998).
Conclusion
With the rising demand for market intelligence and the increasing number of survey
researchers, quantitative survey design presents serious challenges that require a strong
commitment to the positivist principles throughout the research process. Positivist paradigm
assumes the measurability and impartiality of social phenomena. It also infers that knowledge
presents concrete characteristics, including validity, reliability, and generalizability, which
cannot be ignored. Therefore, survey researchers use statistical analyses to minimize potential

weaknesses of their instruments and prevent findings from biases. However, it has been argued
throughout the literature that significant challenges can still emerge from errors, human factors,
or both. Although survey research is daunting, it continues to be popular because researchers find
it to be cost effective, efficient to collect data, and appropriate to answer specific research
questions. While the issues of quantitative survey research that have been discussed throughout
this paper can be easily understood from a scholarly perspective, it is not always the case within
the business community, where the search for a quick answer usually compromises the scientific
rigor of the methodology. It is recommended that future studies investigate the extent to which
scholar-practitioners could strengthen survey design in business research and the credibility of
this promising profession.

References
Amaratunga, D., Baldry, D., Sarshar, M., & Newton, R. (2002). Quantitative and qualitative
research in the built environment: Application of "mixed" research approach.
International Journal of Productivity and Performance Management, 51(1), 17-31.
Barlett, J. E., Kotrlik, J. W., & Higgins, C. C. (2001). Organizational research: Determining
appropriate sample size in survey research. Information Technology, Learning, and
Performance Journal, 19(1), 43-50.
http://search.proquest.com/docview/219816871?accountid=27965
Bean, A. G., & Roszkowski, M. J. (1995). The long and short of it. Marketing Research, 7(1),
20.
Betts, S. C. (2011). Extending the forer test beyond face validity: An experiential approach to
teaching social science methodology. Academy of Educational Leadership Journal, 15,
143-149. http://search.proquest.com/docview/887021253?accountid=27965
BeVier, C. A., & Roth, P. L. (1998). Response rates in HRM/OB survey research: Norms and
correlates, 1990-1994. Journal of Management, 24(1), 97-117.
doi:10.1177/014920639802400107
Bielefeld, W. (2006). Quantitative research for nonprofit management. Nonprofit Management &
Leadership, 16(4), 395-409. doi:10.1002/nml.117
Burton, L. & Mazerolle, S.M. (2011). Survey instrument validity part I: principles of survey
instrument development and validation in athletic training education research. Athletic
Training Education Journal, 6(1), 27-35.
Carr, L. T. (1994). The strengths and weaknesses of quantitative and qualitative research: what
method for nursing?. Journal Of Advanced Nursing, 20(4), 716-721. doi:10.1046/j.13652648.1994.20040716.x
Churchill, G. A. (1979). A paradigm for developing better measures of marketing constructs.
Journal of Marketing Research, 16(1), 64-73.
Clark, A. M. (1998). The qualitative-quantitative debate: Moving from positivism and
confrontation to post-positivism and reconciliation. Journal of Advanced Nursing, 27,
1242-1249.
Collins, D. (2003). Pretesting survey instruments: An overview of cognitive methods. Quality of
Life Research, 12(3), 229-238. doi:10.1023/A:1023254226592
Cortina, J. M. (1993). What is coefficient alpha? an examination of theory and applications.
Journal of Applied Psychology, 78(1), 98-104. doi:10.1037/0021-9010.78.1.98

Couper, M., & Miller, P. (2008). Web survey methods. Introduction, Public Opinion
Quarterly, 72(5), 831-835.
Cronbach, L. J., & Meehl, P. E. (1955). Construct validity in psychological tests. Psychological
Bulletin, 52(4), 281-302. doi:10.1037/h0040957
Curtin R., Presser S., & Singer E. (2000). The effects of response rate changes on the Index of
Consumer Sentiment. Public Opinion Quarterly, 64, 413428.
Eldabi, T., Irani, Z., Paul, R. J., & Love, P. E. D. (2002). Quantitative and qualitative decisionmaking methods in simulation modelling. Management Decision, 40(1), 64-73.
http://search.proquest.com/docview/212077597?accountid=27965
Emons, W. H. M., Sijtsma, K., & Meijer, R. R. (2007). On the consistency of individual
classification using short scales. Psychological Methods, 12(1), 105-120.
doi:10.1037/1082-989X.12.1.105
Firestone, W. A. (1987) Meaning in Method: The Rhetoric of Quantitative and Qualitative
Research. Educational Researcher, 16 (7), 1621.
Frippiat, D., & Marquis, N. (2010). Web surveys in the social sciences: An overview. Population
(16342941), 65(2), 285-311. Retrieved from
http://ezproxy.library.capella.edu/login?url=http://search.ebscohost.com.library.capella.e
du/login.aspx?direct=true&db=sih&AN=55435229&site=ehost-live&scope=site
Green, S. B., & Yang, Y. (2009). Commentary on coefficient alpha: A cautionary tale.
Psychometrika, 74(1), 121-135. doi:10.1007/s11336-008-9098-4
Green, S. B. & Yang, Y. (2011). Coefficient alpha: A reliability coefficient for the 21st century?
Journal of Psychoeducational Assessment, 29(4), 377-392.
doi:10.1177/0734282911406668
Guba, E. G. & Lincoln, Y. S. (1994). Competing paradigms in qualitative research. Handbook of
qualitative research, 2, 163-194.
Harrison, R. T., Hill, F. M., & Leitch, C. M. (2010). The philosophy and practice of interpretivist
research in entrepreneurship: Quality, validation, and trust. Organizational Research
Methods, 13(1), 67-84. doi:10.1177/1094428109339839
Hox, J. (2007). The influence of advance letters on response in telephone surveys: A metaanalysis. The Public Opinion Quarterly, 71(3), 413-443.
Ihantola, E. & Kihn, L. (2011). Threats to validity and reliability in mixed methods accounting
research. Qualitative Research in Accounting and Management, 8(1), 39-58.
doi:10.1108/11766091111124694
Johnson, R. B., & Onwuegbuzie, A. J. (2004). Mixed methods research: A research paradigm
whose time has come. Educational Researcher, 33(7), 14-26.
doi:10.3102/0013189X033007014
Keeter, S., Miller C., Kohut A., Groves R., & Presser S. (2000). Consequences of reducing
nonresponse in a large national telephone survey. Public Opinion Quarterly, 64, 125
148.
Nunnally, J. C. (1978). Psychometric theory (2nd ed.). New York, NY: McGraw-Hill.
O'Leary-Kelly, S. W., & J. Vokurka, R. (1998). The empirical assessment of construct validity.
Journal of Operations Management, 16(4), 387-405. doi:10.1016/S0272-6963(98)000205
Onwuegbuzie, A. J., & Johnson, R. B. (2006). The validity issue in mixed research. Research in
the Schools, 13(1), 48-63.
http://search.proquest.com/docview/211030483?accountid=27965
Parasuraman, A., Zeithaml, V. & Berry, L. (1988). "SERVQUAL: a multiple-item scale for
measuring consumer perceptions of service quality", Journal of Retailing, 64, 12-40.

Ponterotto, J. G. (2005). Qualitative research in counseling psychology: A primer on research

paradigms and philosophy of science. Journal of Counseling Psychology, 52(2), 126-136.
doi:10.1037/0022-0167.52.2.126
Ross, E. M. (2005). It seemed like a good idea at the time. NBCNEWS.com. Retrieved from
http://www.msnbc.msn.com/id/7209828/ns/us_news/t/it-seemed-good-idea-time/
Rossiter, J. R. (2008). Content Validity of Measures of Abstract Constructs in Management and
Organizational Research. British Journal Of Management, 19(4), 380-388.
doi:10.1111/j.1467-8551.2008.00587.x
Sass, T. L., Mifsud, V. A., Helms, J. E., & Henze, K. T. (2006). Treating cronbach's alpha
reliability coefficients as data in counseling research. The Counseling Psychologist,
34(5), 630-660. doi:10.1177/0011000006288308
Schmeets, H. (2010). Increasing response rates and the consequences in the dutch parliamentary
election study 2006. Field Methods, 22(4), 391-412. doi:10.1177/1525822X10381031
Shuttleworth, M. (2009). Criterion Validity: Criterion validity assesses whether a test reflects a
certain set of abilities. Experiment-resources.com. Retrieved on May 5, 2012 from
http://www.experiment-resources.com/criterion-validity.html
Sijtsma, K. (2009). On the use, the misuse, and the very limited usefulness of Cronbachs alpha.
Psychometrika, 74(1), 107-120. doi:10.1007/s11336-008-9101-0
Terrell, R. S. (2012). Mixed-methods research methodologies. The Qualitative Report, 17(1),
254.
Thompson, B., & Vacha-Haase, T. (2011). Score reliability: A retrospective look back at 12
years of reliability generalization studies. Measurement and Evaluation in Counseling
and Development, 44(3), 159-168. doi:10.1177/0748175611409845
Turocy, P. S. (2002). Survey research in athletic training: The scientific method of development
and implementation. Journal of Athletic Training, 37(4), 174-179.
Bureau of Labor Statistics, U.S. Department of Labor, Occupational Outlook Handbook, 201213 Edition, Survey Researchers. Retrieved on April 21, 2013 from
http://www.bls.gov/ooh/life-physical-and-social-science/survey-researchers.htm
Walsham, G. (2006). Doing interpretive research. European Journal of Information Systems,
15(3), 320-330. doi:10.1057/palgrave.ejis.3000589