A NEW MEASUREMENT SCALE FOR EMPLOYEE ENGAGEMENT:

SCALE DEVELOPMENT, PILOT TEST, AND REPLICATION

CHRISTOPHER H. THOMAS
Department of Management
Northern Illinois University
DeKalb, IL 60115

INTRODUCTION

Kahn (1990, 1992) introduced the concept of personal engagement with work as “the
harnessing of organizational members’ selves to their work roles; in engagement, people employ
and express themselves physically, cognitively, and emotionally during role performances,”
(1990, p. 694). As a heightened personal and emotional investment in one’s job and work duties
that goes beyond satisfaction or commitment, engagement has been linked to beneficial
outcomes for both individuals and organizations (Gubman, 2004; Harter, Schmidt, & Hayes,
2002; Kahn, 1992; Robinson, Perryman, & Hayday, 2004; Salanova, Agut & Piero, 2005).
However, development of the research stream has been hampered by conceptual inconsistencies
and an overall lack of theoretical development. Moreover, it is unclear if engagement is a new,
distinct construct, or if it merely represents an aggregate of other established constructs.
The purpose of this research was to clarify the underlying conceptual structure of
engagement, and was conducted in two phases. Phase 1 consisted of construct validation and
scale development (cf., DeVellis, 2003; Hinkin 1995, 1998), and Phase 2 consisted of a pilot-test
of the new scale, and data collection and analysis from a replication sample.

PHASE 1: CONSTRUCT DEFINITION AND VALIDATION

Although Kahn’s (1990) seminal work explored “discrete moments” of engagement

(1992, p. 343), studying engagement as a psychological state remains consistent with his
framework. Specifically, Kahn described psychological presence as an “experiential state
enabling organization members to draw deeply on their personal selves in role performances,”
(1992, p.321). Extant research supports the contention that engagement, as a state, provides a
complex perspective on the individuals’ relationships with their work that accounts for consistent
patterns of workplace behaviors (Maslach & Leiter, 1997; Schaufeli & Bakker, 2003).
Previous researchers have attempted, with varying degrees of success, to empirically
model engagement as a multidimensional state (May, Gilson & Harter, 2004; Schaufeli &
Bakker, 2003). The main departure of this project from previous research is that engagement was
conceptualized as unidimensional. Specifically, although the consequences of engagement (i.e.,
behaviors) occur in three categories—physical, cognitive, emotional—the state preceding these
behaviors was modeled as unidimensional (Kahn, 1992; May et al., 2004).

Scale Development

Scale development was undertaken to create items to measure engagement as a

unidimensional experiential state that is influenced by elements of the individual’s work context.
This measure would account for both the affective-cognitive components of engagement as
captured by existing instruments, and also for work-role behavioral intentions. Engagement is
not an enduring personality trait that is generalizable across situations. Rather, it is a relatively
stable psychological state. In terms of temporal stability (i.e, malleability), variance in individual
levels of engagement can be caused by changes in the work environment or outside-life
influences. Thus,

Employee engagement is a relatively stable psychological state influenced by interactions

of individuals and their work environment. Engaged employees are characterized by a
readiness and willingness to direct personal energies into physical, cognitive, and
emotional expressions associated with fulfilling required and discretionary work roles.

Semi-Structured Interviews. The preliminary set of scale items was generated based on
insights from existing literature and from information gained through semi-structured interviews.
Individuals occupying diverse occupations from various organizations were selected for these
sessions. Males (n = 9) and females (n = 9) each represented 50% of the sample, and the
interviewees ranged in age from 26 to 65 years with a mean age of 36.5 (SD = 9.9 years).
Engagement was strongly related to the job tasks. Respondents tended to connect
engagement to times when they were performing tasks of personal significance to them. Also,
when respondents had knowledge of tangible results and could quantify their progress toward
providing a high-quality work product they reported being more engaged. Respondents also
linked engagement to times when they were particularly driven to be a noticeable asset to the
firm. Individuals were curious and more creative; they approached their jobs in a thoughtful
manner; they critically examined the elements and details of their tasks; they felt a sense of
competence and were willing to take risks; they were more persistent and industrious than was
normal for them; and they sought to provide high-quality products and services while being good
representatives of their organization. Furthermore, among those who considered themselves to be
generally dutiful, conscientious, or industrious it seemed that these characteristics were
magnified during times of engagement. The consensus was that when the right personal, task,
and contextual elements came together a state of engagement was the result.

Item Generation. Items were created to capture the state of mind described by
interviewees. The newly created items were intentionally worded such that they could be used in
diverse employment and research settings. An initial pool of eighty-one items was reduced to
forty-four after a preliminary examination revealed problems like overly complex wording, being
too industry specific, or other violations of “best-practices” for item structure (cf., Hinkin, 1998).

Expert Opinion. The next step in establishing content validity followed the example of
MacKenzie, Podsakoff, and Fetter (1991). Seven subject matter experts (SMEs) were asked to
classify each item based on the degree to which it appeared to be an appropriate measure of the
intended construct. Items that generated at least 70% agreement were retained. SMEs were also
prompted to provide open-ended feedback. Seventeen items met the initial retention criteria.
Two of these items were rated by at least one expert to be of “low relevance or not at all
relevant,” and consequently were eliminated. Between two remaining items with very similar
focus one was chosen at random for elimination. Improvements in item wording were made
following a supplemental round of consultation with SMEs.
 Content Adequacy. Content adequacy was assessed using the procedure described by
Schriesheim et al. (1993). The Schriesheim et al. method is intended for multidimensional scales,
and had to be amended for the proposed unidimensional scale. To approximate this method, the
definitions and scale items from two distinct, but possibly related, constructs were included
along with the definition and items for engagement. This allowed for analysis of whether
engagement items were perceived as being adequate for measuring the intended construct, as
well as whether the items were perceived as distinct from the other constructs.
The sample consisted of fifty-nine MBA students at a large southeastern university. An
extended matrix of respondent ratings was created for this analysis (cf., Hinkin & Tracey, 1999;
Schriesheim et al, 1993). Four rows of data were generated for each respondent representing
their rating for each item on each of the three constructs being compared and a fourth category of
“None of these / Other.” The sample of 59 respondents thereby created a dataset of 236 rows.
Each item displayed the highest mean value on its intended construct, thus establishing
preliminary evidence that the individual items were perceived as belonging to their designated
construct. As no items had the highest mean value on the “Other” category, it was eliminated
from further consideration. Next, the data were analyzed via exploratory factor analysis using
principal axis factoring. Three distinct factors emerged from the data, with all but one item
loading on its intended factor. The item which loaded incorrectly was a newly created
engagement item. It was subsequently discarded.

PHASE 2: PILOT TEST, VALIDITY ASSESSMENT, REPLICATION

The next step was to administer the new scale, along with measures of other constructs,
to a sample representative of the population of interest, (Hinkin 1995, 1998). This allowed
assessment of both factor structure, and convergent and discriminant validity. An email was sent
to 1,500 randomly selected StudyResponse.com members who met the following inclusion
criteria: U.S. resident, employed full or part-time, and at least 18 years old. The initial sample
included 527 responses. Sixty-one cases were removed due to incomplete data, resulting in a
final sample of 466 responses (31% response rate). The mean age of this sample was 38.0 (SD =
10.8). Respondents indicated an average of 67.9 months (SD = 77.3) in their current job, and
70.8 months (SD = 77.4) with their current organization.

Factor Analysis

Both exploratory and confirmatory factor analyses were used to test and refine the scale.
A split-sample method was utilized on the pilot-test data, such that 200 randomly selected
observations were used for EFA, and the remaining 266 observations composed the CFA sample.

Exploratory Factor Analysis (EFA). EFA was conducted in SPSS using Principal Axis
Factoring with Varimax rotation. The a priori assumption was that the items would generate a
single factor solution. The solution converged after nine iterations, with two extracted factors.
The eigenvalues for these factors were 9.94 and 1.46, respectively. Other than being reverse-
scored, a distinct commonality did not exist among the three items composing the second factor.
Researchers have documented cases where reverse-scored items introduce systematic
error resulting in an artifactual response factor consisting of all negatively-worded items
(Harvey, Bilings, & Nilan, 1985; Schmitt & Stults, 1985). Thus, these three items were removed,
and the fifteen remaining items were reanalyzed. The 15-item EFA converged on the expected
single-factor solution (α= .95). Three items generated relatively low item-total correlations, and
also produced the lowest factor loadings. These items were eliminated and the twelve best-
functioning items were retained final scale development.

Confirmatory Factor Analysis (CFA). Using maximum-likelihood estimation via

LISREL 8.52 (Joreskog & Sorbom, 1996), a model was specified with the twelve items
designated to load on a single factor. The fit of this model was marginal (χ2 = 187.85, df = 54, p
<.01; RMSEA = .10; TLI = .97; CFI = .98; GFI = .89). While TLI and CFI indicated good fit
(Hu & Bentler, 1999; Lance & Vandenberg, 2002), GFI was lower than desired, and RMSEA
fell outside the acceptable upper bound of .08.
The residual covariance matrix was examined to determine the sources of poor fit (Byrne,
1998; Loehlin, 1998). The standardized residuals revealed three sets of problematic item pairs.
In each case, the items-pairs were capturing an overlapping portion of the engagement domain,
thus one item from each pair was eliminated. Having eliminated three items, the remaining nine-
items were analyzed. The fit of this nine-item model improved significantly (χ2 = 35.24, df = 27,
ns; RMSEA = .03; TLI = .99; CFI = 1.00; GFI = .97). Scale alpha was .93.

Convergent and Discriminant Validity. Gerbing and Anderson (1988) refer to this stage
as verification of the external consistency of the measure since the test assures that items
continue to be associated with their prescribed scale when examined among multiple measures.
CFA has become the method of choice for this procedure (Bagozzi, Yi, & Phillips, 1991). The
process is similar to the scale refinement process, but with additional measures included (Hinkin,
1998). The results of this expanded CFA supported the single factor structure of work
engagement, and established evidence of convergent and discriminant validity. All items had
significant loadings and the fit of the model was good (χ2 = 1857.79, df = 725, p < .01; RMSEA
= .08; TLI = .96; CFI = .96; RFI = .93) (Hu et al., 1999; Lance et al., 2002; Millsap, 2002).
Additional constructs were chosen based on the assumption that they would occupy space
within the nomological net of work engagement (e.g., organizational commitment, job
satisfaction, turnover intentions, job meaningfulness, and work alienation). Meaningfulness was
expected to positively influence the degree to which workers become engaged with their jobs. As
expected, these two factors were positively correlated at a high level (r =.64, p < .01). Likewise,
individuals displaying high levels of work alienation would not be expected to be engaged. This
relationship was supported by a significant, negative association (r = -.46, p < .01). In terms of
criterion-related validity, engaged workers would be expected to report higher levels of
organizational commitment, greater job satisfaction, and lower turnover intentions. These
assumptions were supported (roc = .51, p < .01; rjs = .52, p < .01; rti = -.40, p < .01). The
associations indicated strong, predictable relationships, and also supported the distinctiveness of
engagement from existing measures.

------------------------------------------
Insert Table 1 about here
-------------------------------------------

Replication Study. The final step was to gather data from an additional independent
sample for replication purposes (Hinkin, 1998). Three units of a not-for-profit, community-
owned health care system located in the southeastern United States were sampled. Unit managers
were asked to distribute surveys at a weekly meeting. Respondents were informed that
participation was voluntary, and that their responses were completely anonymous. Respondents
were allowed to complete the surveys during working hours and returned them via a self-
addressed stamp envelope. Ninety-eight surveys were distributed and 57 were returned during
the next three weeks. The response rate was 58%.
The replication data further supported the findings from the pilot-test sample. To begin,
the engagement scale generated an alpha value of .89. This value is slightly lower than the alpha
value obtained from the pilot-test, but was nonetheless indicative of internal consistency.
Following the suggestions of Fan and Thompson (2001) regarding scale reliability, a 95%
confidence interval for the alpha value was constructed to ensure that the entire range not only
exceeded the minimally acceptable value of .70 suggested by Nunnally (1978), but also meets his
more stringent recommendation of .80 or greater for applied research (Lance, Butts, & Michels,
2006). The lower bound was .85 and the upper bound was .93.
Meaningfulness was strongly correlated with engagement in both samples (rpilot = .64;
rreplication = .65), as was organizational commitment (rpilot = .51; rreplication = .58). Additional
correlation data supported significant relationships between engagement and several behavioral
outcomes. Specifically, with task performance behaviors (r = .49, p < .01), contextual
performance behaviors (r = .77, p < .01), role innovation (r = .49, p < .01), and positive
emotional displays (r = .55, p < .01).
Various demographic characteristic were examined for significant associations. Previous
engagement research reports that engagement levels are higher for supervisors and managers
than for employees at lower levels of the organization (Robinson et al., 2004; Schaufeli &
Bakker, 2003). The current data supported these findings. Mean-level differences between staff
employees, supervisors, and managers were significant (p < .05). Previous studies report that
demographic differences in engagement are statistically non-significant, or those differences that
do exist have been labeled as “significant, but weak,” or lacking “practical significance,”
(Robinson et al., 2004; Schaufeli & Bakker, 2003, p. 18). The current data followed these
patterns in that males did report slightly higher engagement, and the relationship of engagement
with age was in a positive direction; however there were no statistically significant gender, age,
or racial differences.

DISCUSSION

The number of engagement-related publications indicates a growing interest in

engagement as a means of propelling individuals to make greater contributions to organizational
success. This study bolsters existing engagement research, positions engagement as a distinct
construct within organizational research, and provides new insights to propel future research.
The newly developed scale supports the idea that engagement is a state of aroused, situation-
specific motivation that is correlated with both attitudinal and behavioral outcomes. Individually
the scale items appear to measure separate phenomenon like curiosity, diligence, industriousness,
learning orientation, or even achievement motives. However, the amalgamation of these separate
motivations being activated in concert with one another identifies the state of employee
engagement.

REFERENCES AVAILABLE FROM THE AUTHOR

Table 1
Final 9-item Employee Engagement Scale
1. I am willing to really push myself to reach challenging work goals.
2. I am prepared to fully devote myself to performing my job duties.
3. I get excited thinking about new ways to do my job more effectively.
4. I am enthusiastic about providing a high quality product or service.
5. I am always willing to “go the extra mile” in order to do my job well.
6. Trying to constantly improve my job performance is very important to me.
7. My job is a source of personal pride.
8. I am determined to be complete and thorough in all my job duties.
9. I am ready to put my heart and soul into my work.
Note. Pilot-test sample (N = 266, α = .93); Replication sample (N = 57, α = .89).
Copyright of Academy of Management Annual Meeting Proceedings is the property of Academy of
Management and its content may not be copied or emailed to multiple sites or posted to a listserv without the
copyright holder's express written permission. However, users may print, download, or email articles for
individual use.