INTERNATIONAL JOURNAL OF SELECTION AND ASSESSMENT VOLUME 11 NUMBERS 2/3 JUNE/SEPTEMBER 2003

Web-based Assessment of Call Center Agents:

Development and Validation of
a Computerized Instrument
Udo Konradt *, Guido Hertel and Karin Joder
University of Kiel

This study describes the development and validation of the Call Center Aptitude Test
(C-A-T), an Internet-based multi-method measure for the pre-selection of call center
agents. The C-A-T includes a short biographical form, a cognitive speed test, a personality
inventory, and a multimedia situational judgement test. A total of 327 call center agents
completed the C-A-T and were concurrently assessed by their superiors. In a pilot study
(N 5 151), a prototype was tested and subsequently optimized. In the main study
(N 5 176), the revised and extended version of the C-A-T was reliable (alpha 5 .85) and
showed good convergent and discriminant validity. The concurrent validity (r 5.33) can
be considered to be satisfactory for a computerized pre-selection instrument. Evidence
was also found for high face validity and acceptance in both studies.

Introduction

call center is a central place where customer and

other telephone calls are handled. Typically, a call
center has to handle a considerable volume of calls at the
same time, to screen calls and forward them to someone
qualified to handle them, and to log calls. In the last few
years, the call center sector has shown an enormous
economic increase (Huws, Denbigh and ORegan, 1999).
In 1999, the European market was estimated to be worth
about 7 billion Euros, with a yearly growth rate of 40%,
which constitutes one of the most rapidly growing forms of
employment (Bradshaw, Wood and Delaney, 1999). By
2002, about 1.3% of the total workforce in Europe (i.e. 2
million people) will be employed in call centers. In a similar
way, more than 5 million new jobs have been created in the
USA in this sector since 1990 (Bagnara, 2000). Hence
recruitment is becoming increasingly difficult and expensive, with the cost for any new agent estimated to be about
h5,000 (Bagnara, 2000). One main difference in call center
work is whether the calls are inbound or outbound.
Inbound calls mean that a customer is calling the agent who
is passively waiting for calls. Outbound calling means that
*Address for correspondence: Udo Konradt, Department of Psychology, University of Kiel, Olshausenstr. 40, D-24 098 Kiel, Germany.
Email: konradt@psychologie.uni-kiel.de

184

the agent initiates the calls. Typically, outbound activities

concern sales, market research, fundraising, polling or
opinion surveys, whereas inbound calling concerns mainly
customer services or technical support. While inbound
transactions were more prevalent in the past few years, we
recently see an emerging trend toward outbound activities.
In outbound activities the telephone agent or telephone
operator is becoming a Customer Service Representative
with high qualifications as a consultant, an engineer, a
technician, or an account executive supplying products
and services. As a consequence of this trend, call centers
are faced with an increasing need for highly skilled and
motivated employees.
Moreover, employee turnover in call centers is still
higher than average. In an international call center study in
Germany and the United States, 68% of the participating
companies indicated an extremely high employee turnover
of more than 50% a year (Kaapke and Schmitz, 1998). As a
consequence, several studies have recently investigated the
relationship between working conditions and employee
turnover in call centers (e.g., Isic, Dormann and Zapf,
1999; Metz, Rothe and Degener, 2001; Moltzen and van
Dick, 2002). According to the preliminary results, employee turnover rate in call centers can be explained both by
external and internal factors. Examples of external factors
are stressful working conditions and relatively low average
income. For instance, conflicts with clients during work

r Blackwell Publishing Ltd 2003, 9600 Garsington Road, Oxford OX4 2DQ, UK and
350 Main Street, Malden, MA 02148, USA.

WEB-BASED ASSESSMENT

can lead to a predominance of emotion-work, that is the

need to display or suppress emotions on the job (Dormann,
Zapf and Isic, 2002; Sczesny and Stahlberg, 2000).
Moreover, often call center tasks include service work with
very little decision latitudes, insufficient control over
activities, as well as high time pressure (e.g., Isic, Dormann
and Zapf, 1999; Metz, Rothe and Degener, 2001).
As internal factors, most agents have no or very little
experience with call center work and little cognitive or
emotional skills to cope with the related stressors. Most
agents are semi-skilled workers who are often not prepared
appropriately for their job demands (Bagnara, 2000). It has
been shown that the quantitative job demands (e.g. too
high workload), qualitative job demands (e.g. emotional
conflicts and little feedback from customers), and the lack
of sufficient resources on the job lead to more absenteeism,
declines in performance, and burnout (Maslach, Schaufeli
and Leiter, 2001; Zapf, Seifert, Schmutte and Mertini,
2001). With regard to personnel selection, people with
knowledge, skills, and abilities suitable to cope with those
job requirements should be selected.

185

versions of the same test. Video-based assessment in a

situational judgement test elicited less adverse effects and
more favorable face validity reactions in comparison to a
paper-and-pencil application (Chan and Schmitt, 1997).
The higher acceptance of computerized and web-based
instruments is particularly relevant here because it may
enhance the motivation of individuals to participate in
assessment procedures and lessens their motivation to fake
(Arvey
et al., 1990; Snell, Sydell and Lueke, 1999; Schmitt and
Chan, 1999). Finally, computer applications of assessment
tools might increase the accuracy of data collection due to
feedback loops that can be integrated to check whether a
respondent has completed all questions (Stanton, 1998).
Although the internet is already used in recruitment and
personnel assessment (Bartram, 1999) and is seen by
researchers as a major trend for the future (Jones and
Higgins, 2001), selection instruments that take advantage
of available multimedia elements (e.g. video and audiobased items) and the interactive facilities of the internet are
only slowly emerging (Hanson et al., 1999; for recent
reviews on computer- and web-based personnel selection
see Drasgow and Olson-Buchanan, 1999; Hough, 2000).

Computerized Testing
Computerized testing has received considerable attention
in the last few years with an increasing impact on
companies selection and personnel development systems
(Bartram, 1999; Buchanan and Smith, 1999; Drasgow
et al., 1993). In addition to computer-based testing offline
that can be run on a standard PC without any connection
to computer networks, web-based or online measures
have been developed that have a number of advantages
compared to traditional paper-and-pencil instruments.
First, internet-based measures allow testing at any time
and anywhere. Thus, testing can be extremely cost-effective
and efficient particularly when assessing large samples.
Several studies have shown psychometrical equivalence
between paper-and-pencil and online applications (Buchanan and Smith, 1999; Hertel, Naumann, Konradt and
Batinic, 2002; Stanton, 1998). A meta-analysis comparing
computerized and paper-and-pencil ability tests found high
levels of equivalence (r around .90) for power tests. Only
speed tests showed somewhat lower equivalence (r around
.60; cf. Mead and Drasgow, 1993).
Second, computerized measures can have positive effects
on test reliability because participants sometimes answer
more frankly and truthfully (Hinkle, Sampson and
Radonsky, 1991). Third, relatively high social acceptance scores are reported for computerized tests (Arvey
et al., 1990; Chan and Schmitt, 1997; Chan et al., 1997;
Richman-Hirsch, Olson-Buchanan and Drasgow, 2000).
For example, managers completing a multimedia test on
conflict resolution skills rated the instrument as more face
valid and had more positive attitudes towards this test form
compared to managers who completed paper-and-pencil

r Blackwell Publishing Ltd 2003

Development of the Call Center Aptitude

Test (C-A-T)
The Call Center Aptitude Test (C-A-T) was developed as an
internet-based pre-selection measure for call center agents.
The first screening of applications is part of the usual
selection procedure. This screening is often done in a rather
unsystematic way, based on only a small number of criteria
that often depend on the individual preferences of the
person who is doing the screening. More systematic
screening or pre-selection should improve the reliability
and validity of the selection process and lead to an
improved final decision.
Usually, biographical data of applicants are used in
the pre-selection or entry-level. However, these biographical measures have been criticized for a number of
reasons (Bliesener, 1996; Mumford and Owens, 1987). For
instance, Bliesener (1996) found only a small average
predictive validity of .22 in a meta-analysis. Harvey-Cook
and Taffler (2000) recently confirmed this value, reporting
an inter-temporal validity score (after five years) of .23.
Consequently, it seems worthwhile to increase the predictive power of the instruments by using additional
reliable predictors at an early stage of the job application
process.
The development of the C-A-T was guided by the
following objectives: The instrument should be applicable
at any time and anywhere using a computer. The necessary
time for completion should be short. Users should find
the interface appealing, easy to use and they should meet
ergonomic criteria (see Booth, 1998). After the interface

Volume 11 Numbers 2/3 June/September 2003

186

UDO KONRADT, GUIDO HERTEL AND KARIN JODER

application of the C-A-T, a result file should be produced

automatically that contains crucial information about the
applicant for a follow-up face-to-face personnel interview.
Afterwards, the results of the participant are inserted
following the steps of a multi-modal interview (Schuler
and Moser, 1995), which is a semi-standardized interview
technique containing multiple sources of information
about an applicant (self presentation, questions of vocational interests, biographical questions, realistic job preview, and situational questions). Thus, this result file
should also improve the interview process. Finally, the
instrument should enable self selection of applicants and
feature social acceptability (Gilliland, 1993; Schuler, 1993).
Based on the available job analyses in call centers (Bose
and Flieger, 1999; Isic, Dormann and Zapf, 1999; Schuler
and Pabst, 2000; Beron, 2000; Metz, Rothe and Degener,
2001), six dimensions were identified as crucial for call
center agents performance: Mental processing speed,
Ability to communicate via telephone, Social competencies, Persistence, Ability to work under pressure, and
Conscientiousness. In a pilot study, we developed and pretested a prototype version of our multi-method instrument
including a biodata inventory, a cognitive performance test,
and a personality inventory. Below, we briefly describe the
construction and validation of this first C-A-T prototype.
In the main study, a revised version of this pre-test was
evaluated with agents from a different organization. We
expected that the C-A-T would have a reliability of at least
.70 and should have satisfactory internal (factorial,
convergent, and discriminant) validity and external criteria
(concurrent) validity. Additionally, social acceptance data
were collected in both studies. Here, we expected a social
acceptance rate (i.e. agree or fully agree) of more than
75% of the participants. Finally, we expected no differences between inbound and outbound agents because the
C-A-T was developed to assess basic aptitudes rather than
more specialized skills.

PILOT STUDY
Method
Participants and Design
One hundred and fifty-one agents from a middle-sized call
center volunteered to participate in the first study. Since the
focus of this study was mainly on scale development,
no biographical data were collected. However, according
to the HR department of the call center, the participants
met average call center standards regarding age, gender and
education.

Measures
The prototype of the C-A-T was developed as a test battery
with three modules. After a short introduction on how to
use the C-A-T and how to navigate through the modules,

International Journal of Selection and Assessment

the first test module started with a short biodata inventory.

The biodata inventory included four items asking for name,
age, highest school grade and where applicable
profession. The second module contained a newly developed cognitive speed test that measured concentration and
mental processing accuracy. In this task, two hundred and
fifty arrows in ten rows pointing in different directions are
presented on the PC monitor and participants had to
identify target arrows that pointed in a certain direction
(e.g. pointing to the left) using their mouse. This task
consisted of two subtasks that were similar in form and
content. Each subtask lasted two minutes. During the
subtasks, the participants task was to point out target
arrows as fast and correctly as possible. The arrows were
presented on a single screen, and no scrolling was
necessary. The third module was a personality questionnaire with 25 items on five scales. The scales were
(1) Communication skills via telephone (e.g., On the
telephone, it is easy for me to assess someones feelings),
(2) Social competence (e.g., It is rather easy for me to
reveal my opinion in an unknown situation), (3) Persistence (e.g., I pursue my goals, even if they are difficult to
realize), (4) Stress resistance (e.g., In case of throwbacks, I
get discouraged), and (5) Conscientiousness (e.g., I try to
fulfill my tasks conscientiously). Self-ratings were answered on 7-point scales, ranging from completely correct
to completely incorrect. Additionally, participants acceptance of the C-A-T was measured with the following
five items: (1) How clear did you find the questions?,
(2) How certain did you feel when you answered the
questionnaire?, (3) How relevant do you find the
questions for your occupational activities?, (4) Did you
find the questions to be obtrusive?, (5) Did you find this
questionnaire to be appropriate for personnel selection?
Participants answered these items on 5-point scales (from
not at all to completely) with varying polarity.
As criteria, detailed achievement appraisals were conducted by agents supervisors. Agents were rated by
superiors on sixty characteristics on a 4-point scale with
different Likert-type ratings. Twenty out of these sixty
assessment ratings were selected that corresponded to the
ability and personality aspects measured by the C-A-T.
These ratings were in turn accumulated to six criteria
variables (one cognitive performance index and five
personality criteria). The internal consistency of the six
criteria varied between .82 and .91 with a mean of
alpha 5 .87 (see Table 1).

Procedure
Participants were personally invited by the call center
manager to participate in the study. He also informed them
about the goals and procedure. The call center agents
completed the C-A-T during their working hours using
their PC in the call center office. Following a brief
introduction, participants first registered by entering a

r Blackwell Publishing Ltd 2003

r Blackwell Publishing Ltd 2003

.25**
.00
.09
.16
.15
.00
.07
.11
.07
.02
.01
.09

(.24)
.39**
.15
.33**
.21*

.04
.01
.13
.11
.17*
.18*

(.66)
.25**
.37**
.19*

.04
.16
.06
.06
.10
.12

(.58)
.30**
.25**

.04
.12
.09
.18*
.19*
.03

(.60)
.27**

10

11

12

.08 (.90)
.16* .51** (.83)
.08
.58** .64** (.82)
.05
.60** .53** .66** (.87)
.01
.55** .38** .59** .68** (.90)
.08
.44** .52** .55** .48** .37** (.91)

(.69)

Notes. Alpha coefficients are presented in the main diagonal.*denotes significance at po.05,**denotes significance at po.01. Reliability of the concentration
ability (mental processing speed) was calculated as split-half-coefficient.

3.1 (.5)
3.0 (.5)
3.1 (.6)
2.9 (.6)
2.9 (.5)
3.3 (.5)

Criteria Rated by Superiors

7. Learning skills (2)
8. Communication skills (3)
9. Social skills (3)
10. Self-motivation skills (2)
11. Stress resistance (3)
12. Conscientiousness (2)

1
(.87)
.21*
.03
.02
.03
.07

M (SD)

Predictors Measured with C-A-T

1. Concentration ability
52.0 (10.0)
2. Communication skills (3)
5.5 (.7)
3. Social competence (4)
5.5 (.9)
4. Persistence (3)
5.7 (.8)
5. Stress resistance (5)
5.5 (.9)
6. Conscientiousness (5)
6.4 (.5)

Variables (Number of Items)

Table 1. Intercorrelation and Internal Consistencies in the Pilot Study (N 5 151).

WEB-BASED ASSESSMENT
187

Volume 11 Numbers 2/3 June/September 2003

188

UDO KONRADT, GUIDO HERTEL AND KARIN JODER

personal password. Then, participants completed the three

modules of the C-A-T and the acceptance items. The
whole procedure took about 11 minutes (M 5 10.95,
SD 5 3.58).

Results
An exploratory factor analysis (Principal Component
Analysis, Varimax rotation) on the personality test data
of Module 2 was conducted to explore the underlying
dimensions of the newly developed measure. This analysis
revealed seven factors, explaining a total variance of
61.9%. Thus, the results of the factor analysis were not
consistent with the planned five factor structure, suggesting
further adjustments of the developed scales.
The intercorrelations between the cognitive ability
(concentration) measure, the five personality factors, and
the six criteria are shown in Table 1. All personality factors
were significantly related at least at the one percent level
(range between .15 and .39; all po.01), but were
uncorrelated with the cognitive performance measures
(all rso.11, n.s.). With regard to the alpha reliabilities,
only the cognitive ability (concentration ability) measure
(alpha 5 .87; split half) met a satisfactory level of .70.
Except for the Telephone communication skills (alpha 5 .24), reliabilities for the other personality scales
ranged between .59 and .69. Moreover, a ceiling effect for
Conscientiousness occurred (M 5 6.44; SD 5 .48). The
multiple regression that was used to calculate concurrent
validity yielded a coefficient of Rct 5 .20 for the test
battery (corrected Rct 5 .22).
A factor analysis of the acceptance items revealed a twofactor solution with a first factor Appropriateness (Alpha 5
.52) and a second factor Comprehension (Alpha 5 .41).
Overall acceptance, expressed by the mean of all items,
exceeded the chosen threshold of 75% agreement. Acceptance assessment was unrelated to the cognitive ability
measure (r 5 .02, n.s.) but was significantly related to the
aggregated mean personality score (r 5.26, po.001),
indicating that the better the test performance was, the
higher was the acceptance of the test.

Discussion
The pilot study revealed several flaws and some strengths
of the prototype. First of all, the expected factor structure
could not be found in the personality test module,
indicating insufficient construct validity. This had to be
addressed through further adjustment work. Moreover, the
criteria-based validity was low. One explanation may be
that the criteria had already been collected several months
before the predictors were measured, and thus may have
changed over time. Consequently, the validity of the
criteria might have been impaired. For this reason, we
decided to collect concurrent criteria in our main study.
Results showed that the reliabilities of the scales with a
notable exception of the cognitive speed test module were
not satisfactory. However, acceptance of the C-A-T was
fairly high when compared with the acceptance scores for
personality tests and for cognitive performance tests that
have been found in other studies (e.g. Rynes and Connerley,
1993).
Another limitation of the pilot study was that the time of
the day when participants were tested had not been
controlled. However, agents at the end of their shift might
have been considerably more tired compared to the
beginning of the shift, and these variations might have
had a negative effect on test performance, especially on
the concentration test. Moreover, the test battery was
applied in the call center office which may have lead to
disturbances, noise, and/or interruptions by colleagues.
As a first step in our revision of the measure we decided
to re-run the task analysis. Therefore, we conducted
semi-structured in-depth interviews with three call center
managers. Following these results, two constructs
(Ability to communicate on the telephone and Persistence)
were changed and items were renamed. The final five
dimensions of the personality module, their definitions and
sample items are presented in Table 2. Furthermore,
a forced-choice situational judgement test with multimedia
elements was added to the C-A-T. This situational
test consisted of four typical and critical working situations
including an impolite, a verbose, an incomprehensible
and a claiming caller. The data from this test were not

Table 2. Definitions and Sample Items for the Dimensions of the Personality Questionnaire Module included in C-A-T.
1. Willingness to learn: A tendency to be open to training and to enjoy getting into new subjects. Sample item:
I am fascinated by complex subjects.
2. Social competence: A tendency to act socially skilled and to respond cleverly to customers. Sample item:
It is rather easy for me to reveal my opinion in an unknown situation.
3. Self-motivation: A tendency to motivate oneself without external incentives. Sample item: When I dont succeed,
I increase my efforts.
4. Stress resistance: A tendency to cope with time pressure in an efficient manner. Sample item: When I have to work
to a deadline under pressure, I perform less than usual.
5. Conscientiousness: A tendency to be reliable and to meet time-limits as agreed. Sample item: Responsible tasks
are important to me.

International Journal of Selection and Assessment

r Blackwell Publishing Ltd 2003

WEB-BASED ASSESSMENT

used as quantitative predictors but for qualitative purposes.

After test completion, the answers of the participants
were noted on the C-A-T result file/sheet so that they
could be further examined during the subsequent multimethod selection interview (cf. Schuler and Moser,
1995).

MAIN STUDY
Method
Participants and Design
One hundred and seventy-six agents (64 females, 112
males) working in a mid-sized call center (different
to the one in the pilot study) participated voluntarily
in the study. On average, participants were 26.3 years
old (SD 5 3.98). Participants were assessed by four supervisors (1 female, 3 males, on average 26.0 years old,
SD 5 1.41).

Measures and Procedure

The modules provided by C-A-T were conducted using the
revised instrument, consisting now of four modules
(biodata inventory, cognitive speed test, personality inventory, situational judgement test). Acceptance of the
C-A-T was measured as in the pilot study. The agents
also rated their amount of outbound activities. Consistent
with the categorization of four call center managers, agents
who performed less than 30 percent outbound activities
were categorized as inbound agents (n 5 158), while
agents performing more than 30 percent outbound
activities were categorized as outbound agents (n 5 18).
The C-A-T procedure was similar to the pilot study but
was extended by the situational questions with multimedia elements so that completion now took about
16 minutes (M 5 16.4; SD 5 2.54). In order to control
for fatigue effects of the participants, the day-time of the
C-A-T completion was recorded in terms of hours on
the job. The test was carried out in a nearby training
area in which up to eleven agents could work at the same
time.
The superiors assessments of the agents were collected
using a short questionnaire that contained 7-point scales
(ranging from disagree strongly (1) to agree strongly (7))
for the five personality constructs and the cognitive
construct. Each agent was rated by two to four
superiors. Overall, 536 ratings were completed. Thus,
each superior rated 134 agents on average. The interrater
agreement in the performance ratings was assessed by
the T-index (Tinsley and Weiss, 1975) and varied
between .75 and .93 for the constructs, indicating
satisfactory reliability (see Table 3). An overall rating by
superiors was calculated by the mean of z-transformed
single ratings.

r Blackwell Publishing Ltd 2003

189

Results
Prior to confirmatory factor analyses (CFA), item characteristics were assessed. Five items were removed due
to an unacceptable discriminatory power. A CFA was
then used to test the fit of the validation sample data to
the underlying five factor model. Evidence that the model
fits the data acceptably is provided by the RMSEA value
of .06 which meets the desired cutoff exactly (cf. Hu
and Bentler, 1999). Similarly, the goodness of fit index
(GFI 5 0.86; adjusted GFI 5 0.82) and the normed fit
index (NFI 5 0.71) indicated a good fit to the model.
However, the Tucker-Lewis Index result of .83 is under
the recommended threshold of .95, denoting satisfactory
model fit.
Table 3 presents the alpha reliabilities and the intercorrelations between the constructs. The reliabilities of
Social competence, Self-motivation, and Stress resistance
ranged between .59 and .68 and did not meet the level of
.70 required for the internal consistency reliability of personality measures. However, the relatively low reliability
seems to be mainly due to the small scale sizes. Following
the Spearman Brown prophecy formula, an extension on
eight items for each scale would lead to reliabilities
between rtt 5 .70 and rtt 5 .86. The overall reliability of
the test battery was r 5.85 (rtt 5 .92 after scale extension).
As shown in Table 3, the factors were highly correlated,
but do not suggest an underlying single factor model. All
five personality constructs showed significant inter-correlations (po.01), but were uncorrelated with the concentration score indicating convergent and discriminant
validity. With the exception of Stress resistance, no
difference was found between inbound and outbound
activity. The stress resistance was higher for agents who
worked outbound (M 5 5.61; SD 5 .55) compared to those
working predominantly inbound (M 5 5.14; SD 5 .75)
(t 5 2.56; df 5 174; po.01). No differences were found
in the other five constructs (all |t|o1.34, n.s.) or in a
compounded total test score (t 5 1.01; df 5 174; n.s.).
Moreover, there was no significant correlation between
hours in charge before test completion and concentration
ratings (r 5.02, n.s.) or the compounded personality score
(r 5.12, n.s.), respectively. The superior ratings were highly
intercorrelated (each po.01) and were partly marginally
correlated to the predictors.
Forced entry multiple regression analyses were conducted in which the cognitive ability measure and the five
personality factors were used as predictors, and mean
superiors rating was used as dependent variable. This
analysis revealed a significant result (F 5 2.94, df 5 6,
po.01). As revealed by the standardized regression
coefficients (Table 4), significant positive effects were
observed for Social competence (Beta 5 .17, po.05),
Stress resistance (Beta 5 .17, po.05), Conscientiousness
(Beta 5 .19, po.05), and Cognitive ability (Beta 5 .19,
po.05).

Volume 11 Numbers 2/3 June/September 2003

International Journal of Selection and Assessment

.21**
.10
.19*
.19*
.20*
.15
.02
.02
.02
.06
.02
.05

(.70)
.29**
.37**
.34**
.48*

.01
.01
.06
.07
.05
.08

(.68)
.33**
.47**
.38*

.03
.02
.05
.02
.07
.04

.20**
.01
.17*
.10
.14
.05

10

11

12

.13 (.86)
.09
.46** (.93)
.14
.86** .55** (.84)
.11
.79** .68** .77** (.77)
*
.18
.87** .51** .83** .82** (.83)
*
.16
.70** .60** .64** .82** .73** (.75)

(.64)
.41** (.59)
.54** .42** (.72)

Notes. Only correlation coefficients 4.10 are depicted. Alpha coefficients are presented in the main diagonal. Coefficients for the criteria are interrater agreements
(see text).*denotes significance at po.05,**denotes significance at po.01. Reliability of the concentration ability was calculated as split-half-coefficient.

5.0 (1.0)
5.0 (.7)
4.8 (1.0)
4.8 (1.0)
4.5 (1.0)
4.8 (1.1)

Criteria Rated by Superiors

7. Concentration skills (1)
8. Willingness to learn (1)
9. Social skills (1)
10. Self-motivation skills (1)
11. Stress resistance (1)
12. Conscientiousness (1)

1
(.81)
.05
.00
.03
.07
.05

M (SD)

Predictors Measured with C-A-T

1. Concentration ability
46.8 (8.4)
2. Willingness to learn (3)
5.3 (.8)
3. Social competence (4)
5.1 (.9)
4. Self-motivation (3)
5.3 (.8)
5. Stress resistance (5)
5.2 (.8)
6. Conscientiousness (5)
5.7 (.7)

Variables (Number of Items)

Table 3. Intercorrelation and Internal Consistencies in the Main Study (N 5 176).

190
UDO KONRADT, GUIDO HERTEL AND KARIN JODER

r Blackwell Publishing Ltd 2003

WEB-BASED ASSESSMENT

191

Table 4. Multiple Regression Results for the C-A-T predictors.

Variable

Concentration ability
Willingness to learn
Social competence
Self-motivation
Stress resistance
Conscientiousness

0.015
0.086
0.125
0.069
0.152
0.192

SE B
0.006
0.068
0.061
0.073
0.081
0.095

b
0.19
0.11
0.18
0.09
0.17
0.19

t
2.56*
1.26
2.05*
0.95
1.88*
2.02*

Note:*po.05.

For the C-A-T predictors, the overall multiple correlation coefficient of .31 (corrected Rct 5 .33) is statistically
significant at the .001 level, indicating that 9.4 percent of
the overall criteria variance is explained by the C-A-T
predictors. On the other hand, performance was unrelated
to gender, age, education, profession, and the length of time
at work before completing C-A-T (all ro.12).
Finally, a factor analysis of the acceptance ratings
(Principal Component Analysis, Varimax, rotated solution)
revealed two components (loadings 4.67) with the first
factor (Appropriateness) consisting of Item1, Item 2, and
Item 4, and a second factor (Comprehension) consisting
of Item 3 and Item 5. The first factor explained 31.2
percent, and the second factor explained 28.0 percent of
criteria variance. Overall, mean acceptance was 3.85
(SD 5 0.46) and exceeded the expected threshold of 75
percent agreement. Acceptance was found to be independent of the concentration score (r 5.08, n.s.) but correlated
positively with the accumulated personality score (r 5.32;
po.001) and the total test score (r 5.28, po.001).

Discussion
The data analysis of the main study using the revised
instrument revealed good construct validity. The five
personality dimensions which were confirmed by the
confirmatory factor analysis were highly correlated. Apart
from the convergent validity within the personality
module, the personality factors were uncorrelated with
the cognitive concentration measure, indicating discriminant validity (Ackerman and Heggestad, 1997). In
comparison to the pilot study, the psychometric qualities
of the revised scales were significantly improved. Moreover, the ceiling effects observed in the pilot study were
reduced and item difficulties were increased. Four of six
predictors contributed significantly to the performance
criteria showing satisfying concurrent validity. With the
exception of the Stress resistance scale, there were no
differences between inbound and outbound agents. This
indicates a high generalizability of the C-A-T, emphasizing
the function of C-A-T as a general pre-selection instrument.

r Blackwell Publishing Ltd 2003

Finally, social acceptance of the C-A-T was again fairly

high, replicating the positive results of the pilot study.

General Discussion
The aim of this research was to develop and validate a webbased instrument for the pre-selection of call center agents.
The main study supports the utility of this web-based
instrument in pre-selection settings. Overall, a criterionrelated validity of .33 (corrected) was found which is fairly
satisfactory and higher than validity scores usually
observed for biographical data (Bliesener, 1996; HarveyCook and Taffler, 2000). In a recent meta-analysis of
selection instruments, Schmidt and Hunter (1998) reported
a predictive validity score for biodata as a single selection
method of about .35, and for structured interviews of about
.38. Even though a direct comparison between a single
score and a meta-analytic estimate can not be drawn, the
criterion validity of C-A-T can be seen as a first example
that the predictive power of longer and more expensive
instruments can already be achieved by relatively brief
computerized instruments.
C-A-T was developed as a pre-selection tool and may
serve two functions. First, it can be used to select those
applicants that are possibly qualified to perform call center
agent tasks. The second function of C-A-T is to guide a
structured selection interview and may enable interviewers
to focus on contradictory or missing aspects that might
have occurred in the test results. However, such incremental validity can not be enumerated within the design of
the reported studies.
The data on social acceptance yielded results which were
confirmatory in nature. The agents ascribed high face
validity to the instrument and considered it to be helpful,
appropriate and essential. It should be noted that this
high acceptance can not only be ascribed to the inclusion of
multimedia components that usually lead to more positive
reactions (cf., Richman-Hirsch et al., 2000; Shotland,
Alliger and Sales, 1998). The C-A-T already received
positive reactions in the pilot study with no multimedia
elements. Instead, it seems that other aspects of C-A-T, such
as transparency, fairness, and perceived relevance of the

Volume 11 Numbers 2/3 June/September 2003

192

UDO KONRADT, GUIDO HERTEL AND KARIN JODER

instrument to job-related issues might have contributed to

the rather positive evaluation.
The current study has several limitations. First, predictor
and criteria were concurrently collected with agents who
were already on the job. However, the resulting reduction
in systematic variance might lead rather to an underestimation of the predictive validity of C-A-T, thus creating
a more conservative test. However, further replications are
desired to address the questions of predictive and incremental validity of the C-A-T in addition to the concurrent
validity that has been demonstrated in this study. Such
follow-up research should also include cross-validation
approaches. Second, the reactions of participants were only
measured for the whole instrument. Thus, no information
is available about possible differences in reactions across
the four different C-A-T modules. Finally, we merely
measured two aspects of social acceptability while aspects
of participation and perceived feedback were not included.
Additional research should also examine extended aspects
of the selection process.
In summary, this study provides evidence that the
aptitude of call center agents can be predicted with a
time-efficient, web-based multi-modal instrument, both for
inbound and outbound activities. The predictive validity is
satisfactory, particularly in light of the short completion
time of the instrument. Moreover, the instrument was
positively perceived by the users. As far as we know from
the current literature, the C-A-T is the first web-based
assessment tool that has been developed following clear
criteria and that has been validated in real contexts. Despite
minor psychometric weaknesses in the construct validity,
the Call Center Aptitude Test proved to be a suitable
instrument to assess the basic aptitude of call center agents.
In spite of the increasing technological opportunities
connected with computers and the Internet, empirically
developed and validated web-based personnel selection
tools are still rare. This paper can be considered as one of
the first empirical approaches to explore the potential
efficiency of such web-based selection instruments, yielding promising results. We hope that our research might
initiate more research on the psychometric quality and utility
of web-based personnel selection instruments in the future.

References
Ackerman, P.L. and Heggestad, E.D. (1997) Intelligence, Personality, and Interests: Evidence for overlapping traits. Psychological Bulletin, 121, 219245.
Arvey, R.D., Strickland, W., Drauden, G. and Martin, C. (1990)
Motivational components of test taking. Personnel Psychology,
43, 695716.
Bagnara, S. (2000) Towards Telework in Call Centres. Report
produced with the support of the European Commission.
Brussels [http://www.euro-telework.org]
Bartram, D. (1999) Internet Recruitment and Selection: Kissing
Frogs to find Princes. International Journal of Selection and
Assessment, 8, 261274.

International Journal of Selection and Assessment

Beron, E.-M. (2000) Personalauswahl von Mitarbeitern fur den

Einsatz in Call Centern [Personnel selection of employees in call
centers] Langenburg: CCP Verlag.
Bliesener, T. (1996) Methodological moderators in validating
biographical data in personnel selection. Journal of Occupational and Organizational Psychology, 69, 107120.
Booth, J.F. (1998) The User Interface in Computer-based Selection
and Assessment: Applied and Theoretical Problematics of an
Evolving Technology. International Journal of Selection and
Assessment, 6, 6182.
Bose, B. and Flieger, E. (1999) Call-CenterMittelpunkt der
Kundenkommunikation: Planungsschritte und Entscheidungshilfen fur das erfolgreiche Zusammenwirken von Mensch,
Organisation, Technik [Call CentersFcenter of customer
communication. Planning steps and decision guidelines on the
successful collaboration of humans, organization, and technology] Braunschweig: Vieweg.
Bradshaw, D., Wood, S. and Delaney, J. (1999) Next Generation
Call Centres: CTI, Voice and the Web. London: Ovum Ltd.
Buchanan, T. and Smith, J.L. (1999) Using the Internet for
psychological research: Personality testing on the World Wide
Web. British Journal of Psychology, 90, 125144.
Chan, D. and Schmitt, N. (1997) Video-Based Versus Paper-andPencil Method of Assessment in Situational Judgment Tests:
Subgroup Differences in Test Performance and Face Validity
Perceptions. Journal of Applied Psychology, 82, 143159.
Chan, D., Schmitt, N., DeShon, R.P., Clause, C.S. and Delbridge, K.
(1997) Reactions to Cognitive Ability Tests: The Relationships Between Race, Test Performance, Face Validity, and
Test-Taking Motivation. Journal of Applied Psychology, 82,
300310.
Dormann, C., Zapf, D. and Isic, A. (2002) Emotionale Arbeitsanforderungen und ihre Konsequenzen bei Call Center-Arbeitsplatzen [Emotional task demands and their consequences on call
center workplaces]. Zeitschrift fur Arbeits- und Organisationspsychologie, 46, 201215.
Drasgow, F. and Olson-Buchanan, J.B. (Eds.) (1999) Innovations in
Computerized Assessment. Mahwah NJ.: Erlbaum.
Drasgow, F., Olson, J.B., Keenan, P.A., Moberg, P. and Mead, A.D.
(1993) Computerized assessment. Research in Personnel and
Human Ressources Management, 11, 163206.
Gilliland, S.W. (1993) The perceived fairness of selection systems:
An organizational justice perspective. Academy of Management
Review, 18, 694734.
Hanson, M.A., Borman, W.C., Mogilka, H.J., Manning, C. and
Hedge, J.W. (1999) Computerized assessment of skill for a highly
technical job. In F. Drasgow and J.B. Olson-Buchanan (eds),
Innovations in Computerized Assessment (pp. 197220).
Mahwah, NJ: Erlbaum.
Harvey-Cook, J.E. and Taffler, R.J. (2000) Biodata in professional
entry-level selection: Statistical scoring of common format
applications. Journal of Occupational & Organizational Psychology, 73, 103118.
Hertel, G., Naumann, S., Konradt, U. and Batinic, B. (2002) Person
assessment via Internet: Comparing online and paper-and-pencil
questionnaires. In B. Batinic, U. Reips and M. Bosnjak (eds),
Online Social Sciences (pp. 115133). Berlin: Hogrefe.
Hinkle, J.S., Sampson, J.P. Jr. and Radonsky, V. (1991) Computerassisted versus paper-and-pencil assessment of personal problems in a clinical population. Computers in Human Behavior,
7, 237242.
Hough, L.M. (2000) Personnel selection: Looking Toward the
Future Remembering the Past. Annual Review of Psychology,
51, 631664.
Hu, L. and Bentler, P.M. (1999) Cutoff criterion for fit indexes in
covariance structure analysis: conventional criteria versus new
alternatives. Structural Equation Modeling, 6, 155.

r Blackwell Publishing Ltd 2003

WEB-BASED ASSESSMENT

Huws, U., Denbigh, A. and ORegan, S. (1999) Virtually there: the

evolution of call centres. Institute for Employment Studies,
University of Sussex.
Isic, A., Dormann, C. and Zapf, D. (1999) Belastungen und
Ressourcen an Call-Center-Arbeitsplatzen [Stressors and resources of call center jobs]. Zeitschrift fur Arbeitswissenschaft,
53, 202208.
Jones, J.W. and Higgins, K.D. (2001) Megatrends in Personnel
Testing: A Practitioners Perspective. Association of Test
Publisher [http://www.testpublishers.org/journal03.htm]
Kaapke, A. and Schmitz, M. (1998) Staffing, Health & Safety
conditions. [www.eurocallcentre.com/img/PARTII.pdf].
Maslach, D., Schaufeli, W.B. and Leiter, M.P. (2001) Job burnout.
Annual Review of Psychology, 52, 397422.
Mead, A. and Drasgow, F. (1993) Equivalence of computerized and
paper-and-pencil ability tests: A meta-analysis. Psychological
Bulletin, 144, 449458.
Metz, A.-M., Rothe, H.-J. and Degener, M. (2001) Belastungsprofile von Beschaftigen in Call Centern [Strain profiles of call
center agents]. Zeitschrift fur Arbeits- und Organisationspsychologie, 45, 124135.
Moltzen, K. and van Dick, R. (2002) Arbeitsrelevante Einstellungen
bei Call Center Agenten: Ein Vergleich unterschiedlicher Call
Center Typen [Work-related attitudes among call center agents:
A comparison of different types of call centers]. Zeitschrift fur
Personalpsychologie, 1, 161170.
Mumford, M.D. and Owens, W.A. (1987) Methodology review:
Principles, procedures and findings in the application of background data measures. Applied Psychological Measurement, 11,
131.
Richman-Hirsch, W.L., Olson-Buchanan, J.B. and Drasgow, F.
(2000) Examining the impact of administration medium on
examinee perceptions and attitudes. Journal of Applied Psychology, 85, 880887.
Rynes, S.L. and Connerley, M.L. (1993) Applicant reactions to
alternatives selection procedures. Journal of Business &
Psychology, 7, 261279.

r Blackwell Publishing Ltd 2003

193

Schmidt, F.L. and Hunter, J.E. (1998) The validity and utility of
selection methods in personnel psychology: Practical and
theoretical implications of 85 years of research findings.
Psychological Bulletin, 124, 262274.
Schmitt, N. and Chan, D. (1999) The status of research on applicant
reactions to selection tests and its implications for managers.
International Journal of Management Reviews, 1, 4562.
Schuler, H. (1993) Social Validity of selection situations: A
concept and some empirical results. In H. Schuler, J. Farr and
M. Smith (eds), Personnel Selection and Assessment: Individual
and Organizational Perspectives (pp. 1126). Hillsdale, NJ:
Erlbaum.
Schuler, H. and Moser, K. (1995) Die Validitat des multimodalen
Interviews [Validity of the multimodal interview]. Zeitschrift fur
Arbeits- und Organisationspsychologie, 39, 212.
Schuler, H. and Pabst, J. (2000) Personalentwicklung im CallCenter der Zukunft: Fluktuation verhindern, Mitarbeiter langfristig binden [Personnel development in call centers of the
future: Prevention of employee turnover, committing employees
on a long term basis] Neuwied: Luchterhand.
Sczesny, S. and Stahlberg, D. (2000) Sexual harassment over the
telephone: occupational risk at call centers. Work & Stress, 14,
121136.
Shotland, A., Alliger, G.M. and Sales, T. (1998) Face Validity in the
Context of Personnel Selection: A Multimedia Approach.
International Journal of Selection and Assessment, 6, 124130.
Snell, A.F., Sydell, E.J. and Lueke, S.B. (1999) Towards a theory of
applicant faking: Integrating studies of deception. Human
Resource Management Review, 9, 219242.
Stanton, J.M. (1998) An empirical assessment of data collection
using the Internet. Personnel Psychology, 51, 709725.
Tinsley, H.E.A. and Weiss, D.J. (1975) Interrater reliability and
agreement of subjective judgments. Journal of Counseling
Psychology, 22, 358376.
Zapf, D., Seifert, C., Schmutte, B. and Mertini, H. (2001) Emotion
work and job stressors and their effects on burnout. Psychological Health, 16, 527545.

Volume 11 Numbers 2/3 June/September 2003