P1: GCO

Journal of Science Education and Technology pp382-jost-367269 February 26, 2002 21:19 Style file version Oct. 23, 2000

Journal of Science Education and Technology, Vol. 11, No. 2, June 2002 (°
C 2002)

Online Data Collection

Neal W. Topp1 and Bob Pawloski1,2

Online data collection is becoming an essential and efficient tool for evaluators, researchers,
and other educators. This paper touches on elements of the rather short, but eventful history
of online data collection. A brief review of the current literature is presented, followed by a list
of pros and cons to be considered when stepping into online surveying. Finally there is a brief
look at what makes the online survey run—a database connected to the Internet. For those
who would prefer, alternatives to creating or hosting your own online surveys are offered. The
paper closes with contemplation toward the future of online data gathering.
KEY WORDS: online survey; Web survey; online data collection; technology-enhanced evaluation.

FROM PAPER TO E-MAIL
TO WEB-BASED FORMS
In years past, data collection methods for re-
searchers have included oral interviews, hard copy
surveys, and electronically scanned bubble sheets, to
name a few. More recently it would be difficult for any-
one to have been a cyber-citizen on the World Wide
Web3 very long and not have stumbled onto, or maybe
even replied to, at least one online survey or poll. With
the advent of new technological tools, collecting data
online provides several new opportunities, as well as
a few new challenges, for the researcher.
The first surveys over the Internet were predom-
inantly e-mail forms. Limited to text-only, as e-mail
was in the early days, a respondent would usually type
an “x” in between two brackets (i.e. [x]) in the reply,
replacing the standard “bubble” on a hard copy sur-
vey. Additionally, the response to open-ended ques-
tions included some convention for distinguishing the
response from the question. The type of convention
was up to the survey author and was usually dictated
1 University of Nebraska at Omaha, KH 110, 6001 Dodge St.,
Omaha, Nebraska 68182.
2 To whom correspondence should be addressed; e-mail: bob
pawloski@mail.unomaha.edu
3 World Wide Web (WWW) or Web is a browsing system that pro-
vides access to Internet resources based on hypertext graphics-
capable documents. A cybercitizen of the World Wide Web is a
by the particular e-mail client. The response may or
may not have remained distinguishable after being
passed over the Internet from one e-mail client to an-
other. The data would then be electronically pasted
into a database or spreadsheet for analysis.
Eventually, third party software products like
Lasso, Tango, and even Front Page made it possi-
ble for moderately technically literate researchers
to automatically transfer data from Web forms di-
rectly into a database. Gradually, as this type of user-
friendly software became easier and more available,
it eliminated the need to employ a “CGI4 -scripting
techie” in order to Web-enable surveys for educa-
tional research. Online data collection via the Web
has now become a fairly common tool for professional
evaluators as evidenced by the January 2001 thread:
“Re: Web-based Survey Question” on EvalTalk, the
listserv5 of the American Evaluation Association
(http://www.eval.org, accessed March 2001).
Regardless of the increasing popularity, ease of
use, lower costs, and thus the democratization of on-
line surveying, some advantages of the e-mail ap-
proach over Web-enabled surveys remain, and are
enumerated in Christine Smith’s article, “Casting the
4 CGI (Common Gateway Interface) is a system that allows server
software to call on external programs, such as a database applica-
tion. CGI is usually programmed in a language such as PERL or
C++.
5 A listserv is an electronic mailing list whereby messages can be

frequent user the World Wide Web. shared by a number of subscribers.

173
1059-0145/02/0600-0173/0 °
C 2002 Plenum Publishing Corporation
P1: GCO
Journal of Science Education and Technology pp382-jost-367269 February 26, 2002
174
Net” (1997). These advantages include “facilitative
interaction between survey authors and respondents,
collapsed geographic boundaries, user convenience,
and arguably more candid and extensive response
quality” (Smith, 1997, p. 1). E-mail was the preferred
method over the Web in a study by Sheehan and Hoy
(1999). They utilized Four11.Com (which has since
been engulfed by Yahoo.Com) to generate a nation-
ally representative sample of Internet users. The au-
thors acknowledged the flexibility and user friend-
liness of Web-based forms, but preferred to avoid
the broad base response that occurs when users “self-
select” themselves into a survey sample, making it to
track response rates.
Other research indicates that elapsed time from
setup to delivery is shorter with e-mail surveys
(Bowers, 1999), and that response rates for e-mail
and Web-based surveys are becoming close to equal
(Smith, 1997; Stanton, 1998). A remarkably com-
prehensive compilation of research of online survey
methods can be found at the site of the Research on
Internet in Slovenia (RIS) Project (WebSM, 2001).
Among the RIS findings in the literature and from
RIS’s own studies, it has been found that Web surveys
that invite participation by e-mail have a lower rate of
return than surveys that invite participation by ground
mail. It is suggested that respondents tend to regard
e-mail invitations for survey response as “SPAM” or
“junk e-mail.”
The Office of Internet Studies (OIS), located in
the Teacher Education Department at the University
of Nebraska at Omaha, began doing online data col-
lection in 1994. OIS’s history of conducting online
research parallels the evolution of online surveying.
OIS’s first surveys involved research on Nebraska
K-12 teachers’ use of the computer and the Internet.
During the 1994–95 academic year, surveys were
e-mailed to a specific population of Nebraska K-12
teachers who had completed training on how to use
the Internet. This training was the prerequisite to
these teachers each receiving an Internet e-mail ac-
count. The 1995–96 survey was delivered in a simi-
lar manner with a cumulative population. There were
several factors affecting the rate of return these first
2 years. Most observed difficulties involved people
not using their e-mail, still not knowing how to use
e-mail, not having a computer, not being connected to
the Internet, and having their e-mail address changed
for one reason or another.
By the end of 1996, OIS had the technology
in place to deliver the next 2 years of teacher sur-
veys via a Web page. Responses were then submitted
21:19 Style file version Oct. 23, 2000
Topp and Pawloski
automatically to a FileMaker Pro database residing
on the OIS Web Server. Tango by EveryWare was the
out-of-the-box CGI software used to create the sur-
vey form and link it online. Responses to the survey
in this third year (1996–97) more than doubled. The
following year (1997–98) the migration was made to
Version 4 of FileMaker Pro with its new Web Com-
panion feature. Moreover, this time the Web surveys
were prefaced by a request to participate to each of
the 19 Educational Service Unit (ESU) technology
specialists, who in turn sent the e-mail announce-
ment to every teacher who had an e-mail account
within their jurisdiction. This announcement also in-
formed respondents of the survey’s purpose and URL
(address on the Web). This sample included essen-
tially all of the teachers in the state of Nebraska
who had e-mail accounts at the time (a majority).
Unfortunately, whether everyone used their e-mail,
or if addresses had changed as they often do (GVU8,
1997), was relatively impossible to determine within
the scope of this study.
In 2000 and 2001 Dr. Neal Topp teamed with
Dr. Elliott Soloway of the University of Michigan
to conduct the “Snapshot Survey of Nebraska”
that surveyed educational technology practice in
Nebraska. It is worth noting that this service is avail-
able at no cost to schools or school districts (http://
www.snapshotsurvey.org; accessed August 2001). The
invitations to participate in this survey were again
sent through the ESU technology specialists. The re-
sponses dropped from 3100 in the first year to 1500 in
the second year of the Snapshot Survey of Nebraska.
It can be speculated that teachers are starting to re-
gard such e-mail as SPAM, concurring with the find-
ings of RIS. However, other factors could include the
changing face of e-mail authority in and between the
ESU’s and school districts. The GVU8 WWW Survey
(GVU8, 1997) notes increasing numbers of Internet
users owning more than one e-mail address.
Determining an accurate response rate, along
with encouraging potential respondents to complete
an online survey (Gaddis, 1998; Meehan and Burns,
1997), are important considerations for online data
collection. Meehan and Burns (1997) noted that the
inability to determine the number of people on a list-
serv hinders calculation of accurate response rates.
E-mail solicitations have the same type of problem.
This finding is in line with the OIS experiences in
surveying Nebraska educators. Gaddis (1998) sug-
gests that it is useful to precede any new survey with
an introduction and purpose to elicit participation and
cooperation, even when there was prior knowledge of
P1: GCO
Journal of Science Education and Technology pp382-jost-367269 February 26, 2002
Online Data Collection
the survey. Using this type of mixed media approach,
the online data collector may find a higher response
rate. The solicitation to participate might be through
television, radio, and newspaper advertisements, or
possibly a U.S. mailed scenic postcard or brochure
that teachers may be less likely to throw away.
A well thought out campaign could possibly bring
about a sense of owing something back to their pro-
fession in the minds of the respondents. Jeffrey Pfeffer
suggests, in his book Managing with Power (1992,
p. 106), that building a sense of “reciprocity” is one
of the most successful techniques for gaining compli-
ance and has often been used in traditional surveys
(i.e. a dollar included with the invitation to respond).
Overall, in the evaluation of two United States
Department of Education Challenge Grants6 and
two PT3 Grants,7 the Office of Internet Studies
gained a wealth of experience collecting data online.
Another online data collection technique success-
fully employed by OIS was to monitor and catego-
rize each project’s listservs. These e-mail discussion
groups were established by the Challenge Grants to
give peer and expert support, which could extend
beyond face-to-face staff development experiences.
Growth was shown in the number of questions posted,
as well as the levels of technical sophistication of the
questions, increasing each year over the life of the
grant (Abdouch et al., 1998).
WEB SURVEY ADVANTAGES
Accuracy, Ease, and Speed of Data Entry
Online collection of data has several advantages
in regard to the actual data collecting process. In
essence, the respondents enter the data directly into
the database. The data can then be collected and en-
tered into a statistical package within minutes. Using
relatively flexible Web page layouts, researchers can
more easily ensure accurate data entry, eliminating
error caused by entry of the respondent, by office
staff reading from handwritten forms, or by smudges
6 Challenge
Grant (The Technology Innovation Challenge Grant
Program) is a U.S. Department of Education program that helps
schools meet the educational needs of their students through the
development of new applications and creative ways to use tech-
nology for learning.
7 PT3 (Preparing Tomorrow’s Teachers to Use Technology) is a U.S.
Department of Education grant program that is designed to sup-
port teacher preparation institutions that implement innovative
program improvements to prepare technology-proficient educa-
tors for 21st century schools.
21:19 Style file version Oct. 23, 2000
175
causing scanning errors (Smith, 1997). Stanton (1998)
suggests that the use of pop-ups, radio buttons, and
check boxes enables better control of responses. Tips
to enhance accurate data entry by respondents and
encourage their participation include: (a) keeping the
form short—no more than 4–8 pages (Gaddis, 1998)
or 20 min (Bowers, 1999), and (b) keeping the layout
easy for the eye to follow (Read, 1991).
Mixing of Types of Media
Web-accessed surveys can incorporate sound,
video, and multimedia application products as part
of the items. Researchers can then assess partici-
pants’ interpretations of graphics, art, or music. This
may become important in student assessment, as it
is based more and more on products rather than test
scores (Garrison and Fenton, 1999). The researcher
can provide specific and interesting multimedia port-
folios full of digital artifacts that help the stakeholders
understand the evaluation in a clearer, more meaning-
ful way.
Immediate Results Feedback
During the OIS experience of providing for-
mative evaluation the U.S. Department of Educa-
tion Technology Innovation Challenge Grants, par-
ticipants of multiday teacher workshops were asked
to complete online evaluation forms at the end of
each day. The workshop planning team was then able
to review the evaluations in the evening and makes
changes in the next day’s workshop based on these
electronically searchable data. This type of immediate
turnaround of data is frequently cited as an advantage
(Hays, 1998; Read, 1991; Smith, 1997; Stanton, 1998).
Ease of Access of Data and Results
Online data collection can make annual or sum-
mative reports easily accessible to a broader audience,
in formats that are easy to access and more meaning-
ful to the user. Web-based reports are much easier to
complete when data is in a database, or all of the data
is in digital format, such as audio and video files.
BARRIERS, CHALLENGES,
AND DISADVANTAGES
As stated above, collecting data online has sev-
eral advantages, but of course there are some barriers,
challenges, and disadvantages also.
P1: GCO
Journal of Science Education and Technology pp382-jost-367269 February 26, 2002
176
Equipment and Know-How
In order to collect data online, the evaluator must
have access to the equipment and technical know-how
to accomplish the task. Web servers, database pro-
grams, and technical expertise are all needed, in ad-
dition to the traditional knowledge of an evaluation
office. Passwords and other control techniques, which
are used to ensure the integrity of respondents, re-
quire time, expertise, and expense (Stanton, 1998).
Furthermore, the size, speed, and reliability of servers,
the speed of connection, and the number of simulta-
neous responses must be considered.
In years past, the major barrier to online data col-
lection has been the unavailability of Web-connected
computers to respondents. Several times traditional
hard copy surveys have had to be used because of
lack of computer access by the intended respondents.
This is changing, but the availability of computer Web
access must still be addressed when considering online
data collection.
Situations and Expertise of Respondents
Research of survey taking has shown a demo-
graphic skew, as greater levels of education can re-
duce computer anxiety (Stanton, 1998). Often, a re-
spondent may not have enough computer savvy to
complete the collection instrument, whether it be a
simple form type survey, uploading student project
examples, or completing an online unit plan with
all the needed resources included. Sometimes novice
users are unsure that their responses were accepted,
and they accidentally create multiple or partial re-
sponses (Hays, 1998). Terminal and server lockups
and submission errors have occurred when there are
numerous simultaneous submissions (Smith, 1997).
Accidental or duplicate submissions have occurred
often enough to make “computer anxiety” a real fac-
tor in any researcher’s analysis process.
Reliability of Online Data Collection
Another challenge to online data collection is
ownership of the entered data. With paper/pencil col-
lection, usually the evaluator is assured that the per-
son completing the survey is the “real” owner of the
responses. With online collection, it is possible that
an imposter is completing the form. This is especially
a concern when the respondents complete the form
on their own, without supervision (Smith, 1997). It is
21:19 Style file version Oct. 23, 2000
Topp and Pawloski
also impossible to tell the focus of attention or the
disposition of the respondent (Stanton, 1998).
The question is often posed as to whether the data
is the same when respondents complete surveys on-
line compared to paper/pencil collection. Research in-
dicates that the medium causes significant difference
only on timed cognitive tests (Mead and Drasgow,
1993 as cited in Stanton, 1998). A generalization
can be made that as respondents become more and
more comfortable with technology and the World
Wide Web, the more the online gathered data will
mirror traditional methods of collection. It is worth
remembering that many of us are immigrants to this
technology, while our children are native to it.
In Search of Validity
Validity should always be a concern when using
survey instruments—online or not. Web-based sur-
veys do face a threat in the area of predictive validity
in that the populations are usually biased toward those
who have access to the Internet and/or the inclination
to respond to online surveys.
One way to address content validity could be to
use an online process. Such a process was used in the
OIS development of the Technology Ability Percep-
tion Self-Report Instrument (TAPSI), an online in-
strument. TAPSI served as one facet of the evaluation
of the UNOmaha College of Education’s PT3 Imple-
mentation Grant. In order to go beyond mere “face
validity” (Trochim, 1999) the TAPSI developers uti-
lized an innovative technique to determine whether
the measure is a good reflection of the construct for
which it was designed. During the instrument devel-
opment phase, content experts from across the state
were solicited by e-mail and given the opportunity to
rate the appropriateness of individual scale items via
a Web-based form. The participation of many of these
experts would have been impossible, if it had not been
facilitated by Web-based forms.
A PEEK UNDER THE HOOD
After considering the above, if one decides that
a Web-enabled database survey is the solution, how
can it be created without much technological trauma?
The mechanics of creating an online survey can be as
simple as using “save as” in many applications, such as
Microsoft Word. Researchers are likely already famil-
iar with creating survey documents in a word process-
ing application. It is thus a logical and straightforward
P1: GCO
Journal of Science Education and Technology pp382-jost-367269 February 26, 2002
Online Data Collection
progression to create a Hypertext Markup Language
(HTML) version of the same, since one of the “save
as” options is usually HTML. They can further tweak
an online form with such popular programs as Home-
Page, Front Page, or DreamWeaver, to name only a
few. Many educators and researchers have now been
at least introduced to the design of Web pages, using
graphical WYSIWYG (What You See Is What You
Get) editors.
It is easy to select from among “objects” such
as: (a) check boxes, for multiple responses to a single
question; (b) radio buttons, limited to one response
per question; or (c) text area for an open-ended
response. These can be dragged to the desired position
immediately following the text of the question, for-
matted appropriately, and validated to be easily read
on whatever browser a respondent may use (although
there the format largely depends on what browser
the end user employs). Each of the above mentioned
“objects” in the form must be tied to the appropriate
question in a related database category with the same
fieldname (tip: use lower case, no spaces, and 8 or
21:19 Style file version Oct. 23, 2000
177
less characters). Creating the new database with fields
(categories) to match the questions is also a rela-
tively simple technical task, with the possible pitfall
of forgetting to choose whether each field should be
designated for numeric or text input. The concept of
response coding will come into play, as it is most de-
sirable to have a numerical value to represent vari-
ous responses to questions for data analysis. A “yes”
response might be coded as a “1” and a “no” as a
“0.” One can imagine how a Likert scale may also be
coded—each response for a single question retaining
the same “fieldname,” yet different values for each
different response. Care should also be taken to set
“default responses” so that nonanswers can be coded
properly (Gaddis, 1998).
Possession of a server or a relatively new desktop
computer that has a good connection to the Internet
24 h a day allows the use of FileMaker Pro (on either
STEP BY STEP INSTRUCTIONS TO
WEB-ENABLE A FILEMAKER PRO DATABASE
1. Create the submission form or convert existing
document.
2. Print out the new/converted submission form.
3. Determine database field names and values for
each entry on the form and write them on the
printout.
4. Create the database.
5. Enter the field names.
6. Modify the submission form to include the field
names EXACTLY as they exist in the database.
7. Add the required FileMaker Form tags.
8. Create two additional HTML pages - a
“Successful
9. Submission” page and an Error page.
10. Test them!
a Mac or PC system) to serve the survey to the World
Wide Web. In the past this required a CGI program-
mer or possibly one of the third party solutions ref-
erenced earlier. Some Web editors now work with
FileMaker, through tutorials or “wizards,” to correctly
link the form to the proper database and response
pages. This can be very helpful when it comes to the
final important task, which is to test the survey to
make sure submission of a form successfully creates a
“new record” in the database on the server.
ALTERNATIVES AND THE FUTURE
This paper is merely a snapshot of the current
state of this technology. Already there are several
available services that guide the developer through
the entire process of creating, posting, hosting,
P1: GCO
Journal of Science Education and Technology pp382-jost-367269 February 26, 2002
178
and compiling results. One such service is Survey-
Builder.com, which can manage the entire process
over the Web, charging fees from $5 to $15 per person
surveyed. SurveySolutions for the Web software by
Pereseusdevelopment.Com is available for around
$179 off the shelf. Either of these solutions can provide
an alternative to having a local server (Lake, 1998).
Downloadable survey applications, which incor-
porate the questionnaire into an executable file, can
provide an even more interactive environment. These
can include complex skip patterns and take advantage
of Windows-based controls, but they are expensive
and take time to distribute. Furthermore, sophistica-
tion of users and size of connection to the Internet
can be serious drawbacks (Bowers, 1999).
Whether the choice is to outsource or to use a
local server with the latest software, the near future
will undoubtedly make the online process simpler,
with more features, better security, and higher lev-
els of data integrity. Analysis programs such as SPSS
will likely be fewer and fewer steps away from the
online participants. We are already starting to see
researchers doing field observations with personal
data assistants (PDA’s), like Palm Pilot or Visor. The
database fields may be created on the desktop and
recorded into these battery-powered hand-held in-
struments that can immediately beam coded obser-
vations, via a wireless Internet module in the PDA, to
a main database on a computer virtually anywhere on
the World Wide Web. Imagine that.
REFERENCES
Abdouch, R., Grandgenett, N., Topp, N., Ostler, E., Pawloski, B.,
Timms, M., and Peterson, J. (1998). The Community
21:19 Style file version Oct. 23, 2000
Topp and Pawloski
Discovered: The Search for Meaning Through the Integration
of Art and Technology in K-12 Education. Evaluation Progress
Report No. 4. (ED436470)
Bowers, D. K. (1999, Winter98/Spring99). FAQ on online research.
Marketing Research 10: 45–49.
Gaddis, S. E. (1998, June). How to design online surveys. Training
and Development 52: 67–72.
Garrison, S., and Fenton, R. (1999, July–August). Database driven
web systems for education. Educational Technology 39: 31–
37.
GVU8. (1997). GVU’s Eighth WWW User Survey. Georgia Tech-
nical Institute’s Graphics, Visualization and Usability Center.
Retrieved August 30, 2001, from http://www.gvu.gatech.edu/
user surveys/survey-1997-10/
Hays, R. (1998, April 13). Internet-based surveys provide fast re-
sults. Marketing News 32: 13.
Lake, M. (1998). Surveying the scene: Take the sting out of in-
formation gathering. Computer Currents. Retrieved August 9,
1999, from http://www.currents.net/magazine/national/1609/
cprc1609.html
Meehan, M. L., and Burns, R. C. (1997, March 24–28). E-mail
survey of a listserv discussion group: Lessons learned from
surveying an electronic network of learners. Paper presented
at the Annual Meeting of the American Educational Research
Association, Chicago, IL. (ERIC Document Reproduction
Service No. ED 411 292)
Read, W. H. (1991, January). Gathering opinion online.
HRMagazine 36: 51.
Sheehan, K. B., and Hoy, M. G. (1999, March). Using e-mail
to survey Internet users in the United States: Methodology
and assessment. Journal of Computer-Mediated Communica-
tion 4: Retrieved August 29, 2001, from http://www.ascusc.org/
jcmc/vol4/issue3/sheehan.html
Smith, C. B. (1997, June). Casting the Net: Surveying an
Internet population. Journal of Computer-Mediated Commu-
nication 3: Retrieved August 29, 2001, from http://jcmc.huji.
ac.il/vol3/issue1/smith.html
Stanton, J. M. (1998, Fall). An empirical assessment of data col-
lection using the Internet. Personnel Psychology 51: 709–
726.
Trochim, W. (1999). The Research Methods Knowledge Base, 1st ed.,
Atomic Dog Publishing, Cincinnati, OH. Retrieved August 29,
2001, from http://trochim.human.cornell.edu/kb/
WebSM. (2001). WebSM: Web Survey Methodology Study. Re-
search on Internet in Slovenia Project website. Retrieved
August 30, 2001, from http://wsm.org