Conceptualising Research Performance (submitted copy)

Pat Bazeley
Research Support P/L
PO Box 2005
Bowral NSW 2576
Australia
Email: pat@researchsupport.com.au

Abstract
In a context of increasing emphasis on academic performance and accountability, data
from a structured survey in which academics elaborated on eight different attributes of
high-performing researchers were used to build a conceptual model of research
performance. From these data, research performance was seen to comprise two basic
components, with six secondary level dimensions and a range of potential indicators.
Four essential (necessary and sufficient) dimensions, relating to the research activity
component of research performance were: engagement, task orientation, research
practice and intellectual processes. Two alternative dimensions (of which at least one is
necessary) relating to the performance, or making research visible, component of
research performance were: dissemination, and collegial engagement. Research
performance was seen to occur within conditions provided by an institutional context
(education and training; opportunity and resources), and to bring about a range of
outcomes (product, impact and reputation).

Keywords: research performance; research assessment; concept analysis; academic

performance; research active status

Published in Studies in Higher Education, 2010, 35(8), 889-903

Background
In the current university environment, scoring well on measured criteria of research
performance establishes our authenticity as researching academics at both institutional
and personal levels, thereby providing capital for the purchase of increasingly scarce
resources (time, money) needed for further research and confirmation of prestige
(Archer, 2008; Bourdieu, 2001). ‘[Determining] which properties are pertinent,
effective and liable to function as capital,’ as ‘the criteria of legitimate membership and
legitimate hierarchy’ is one of the issues arising as a consequence (Bourdieu, p. 11).
Governments in many countries are now basing a significant proportion of universities’
research funding on assessment of performance, consequently efforts to measure
research performance have multiplied in the last decade. Schemes vary with regard to
the unit of assessment (individual, group, department; ‘active’ versus all members); in
what they include as items of assessment; and in their approach, for example, to issues
of assessing quantity versus quality.
Traditional measures of research performance have been based on publication
output (in various configurations), citations as a measure of impact, sometimes some
other assessment of quality of work (e.g., by expert panels, in the UK), and, in some
systems, indicators of the reputation of researchers. Until recently, the Australian
system included recent research income as a major component of its assessment system
(which confuses input with output), with higher degree completions and counts of a
limited range of indicator publication types providing the other components. Its
emphasis on quantity rather than quality created widespread dissatisfaction (Batterham,
2004), with dissatisfaction amongst academics when the scheme was first introduced
providing the initial impetus for this study. A recent attempt to set up an alternative
system which included measures of quality and broader impact as well as volume has
failed to achieve implementation, and the development and implementation of
alternative measures, including the use of a proxy measure of publication quality, is in
progress.
This widespread dissatisfaction with indices of performance, particularly those
that are publication-based, is supported by an extensive literature in which the impact of
irrelevant factors (i.e., things that are not performance related) has been shown to
strongly influence publication success, citation counts and assessments of quality (e.g.,
Creamer, 1998; McNay, 2003; Steele, Butler & Kingsley, 2006). One of the unfortunate
impacts of the current system, for example, is that ‘publication is now viewed as the
objective of research, rather than the dissemination of the knowledge contained within
it’ (Steele, 2004, p. 67). A further complicating factor is the difficulty of assessing non-
traditional outputs from research, which means that those in arts-based and professional
disciplines, and those where publishing is largely book- rather than journal-based, are
especially disadvantaged (Bazeley, 2006a; Steele et al., 2006).
Despite frequent critiques of the appropriateness of and best approaches to
measurement, there has been a marked lack of explicit debate in the academic literature
about what research performance actually means. Expediency, or what is easily
practicable to collect, has been a primary driver in determining what to measure. With
few exceptions (primarily stemming from the work of Bourdieu, 2001), approaches to
assessment of research performance have been driven by empirical data collection
without a theoretical base (Tight, 2004). Goertz (2006) notes that there is not a lot of
attention given to concepts generally in the social science literature and that
operationalisation of a concept has often come before theorisation or conceptualisation,
such that a focus on scaling and measurement distracts from attending to the concept
itself. He argues that both conceptualisation and measurement are important, but that
theory should drive methodology. ‘A concept involves a theoretical and empirical
analysis of the object or phenomenon referred to by the word. A good concept draws
distinctions that are important in the behaviour of the object. The central attributes that a
definition refers to are those that prove relevant for hypotheses, explanations, and causal
mechanisms’ (Goertz, 2006, p. 4). This study therefore attempts to redress this lack of
theoretical development with regard to research assessment by developing a conceptual
model of research performance. Concepts are the building blocks of theory, and theory
should drive measurement. An empirically derived conceptual and theoretical base has
potential, therefore, to contribute to the improvements in the assessment of research
performance.
Åkerlind (2008) has recently worked on developing a general model of what it
means to be a researcher (but not specifically of performance). Synthesising the
localised conceptions of Brew (2001), Bruce, Pham and Stoodley (2004), Bowden,
Green, Cherry and Usher (2005), and others, Åkerlind proposed that key dimensions of
what it meant to engage in research were researcher intentions, research process,
anticipated outcomes, and object of study (research questions/purposes)—to which she
added, from her own data, researcher feelings. She found also in her own study that
researchers talked about fulfilling requirements (sense of duty), establishing themselves,
developing personally and enabling change as four different and progressively more
aware ways of being a researcher. Of particular relevance in the current context were
the anticipated outcomes of research by (and for) researchers located in each of those
four categories of awareness—respectively: concrete products (e.g., publications,
problem solved), academic standing, personal understanding, and benefits to the
community.

Aim/Questions
The purpose of this project was to develop an empirically-based, theoretical concept of
research performance which could then be used to inform thinking about useful
indicators of research performance at a local, if not national level.
It is recognised that indicators at a national level must employ a degree of
expediency—the cost should not outweigh the benefits. At the local level, however, (at
least in Australia) expedient national performance indicators, designed only to compare
whole institutions, are employed inappropriately to assess the ‘research active status’ of
individual academics and as a basis for distribution of funding to departments (Bazeley,
2006a). While the results of this study are designed to inform work potentially leading
to operationalisation and measurement of research performance, that consequential
stage is not developed here.
 Specific questions guiding this foundational (conceptual) part of the study,
therefore, were:
(1) How are high performing researchers described by Australian academics?
(2) What are the dimensions of research performance and how are they structured?

Study design and methods

The overall study, from which this article provides a foundational analysis, employed an
integrated mixed methods design, conducted within an interpretive framework (Bazeley,
2009a, 2009b). It comprised a one-off collection and combination of categorical,
numeric and text-based data, and the iterative use of textual and statistical analysis
procedures using separate, combined, converted and blended forms of the data (Bazeley,
2006b). The analysis reported here relies almost entirely on analysis of the qualitative
data component of the study, and uses an approach to analysis which might best be
described as phenomenographic in so far as it attempts to describe the components and
structure of an experience from the point of view of the reflective conceptions of those
who have that experience (Marton, 1981). Additional comparative analyses and
structural statistical analyses based on the set of codes refined through this initial
interpretive process of concept development will follow in further articles.
Data
The data, gathered via self-administered questionnaire, included:
1. Basic demographic data, including university affiliation, gender, level of
qualification, and level of appointment.
2. Responses to eight open-ended questions, each of which asked respondents to think
about and describe a researcher they knew (or knew of) who demonstrated a
particular attribute. The eight attributes were: ability, quality, satisfaction,
productive, benefit, active, approachable, recognition. It was up to the respondent to
decide whether they thought about and described the same researcher for more than
one of the attributes, or whether they described as many as eight different
researchers. The attributes used were selected on the basis of preliminary data
gathered in the course of conducting and evaluating a research development
program for academic staff in a new university (and generally sit quite well with the
findings of Åkerlind’s [2008] research). The order of the questions was varied for
different respondents, to counter the impact of response bias.
3. Additional quantitative data, rating the relative importance of each attribute in
different contexts, which are not reported here. Respondents also had the
opportunity to add general comments.
Sampling and attributes of respondents
Questionnaires were individually addressed to 2090 teaching academics across all
departments, as listed in the University Calendars, for three universities representing a
cross-section of Australian university types. Responses to the open-ended questions
asking for descriptions of researchers were returned by 295 (some of those listed would
no longer have been employed by the relevant university). ‘Sandstone’ (n=110) is an
older, well-established research university; ‘Greenfield’ (n=82) opened in 1967 as an
innovative, research oriented university; and ‘New’ (n=103) was established as a
university by combining a number of teaching oriented Institutes and Colleges when the
higher education system was unified in 1989.
A higher proportion of those responding were male (n=179, 61%) rather than
female (n =115, 39%). There were 101 in Science (36%), 121 in Social Science (44%)
and 55 from the Humanities (20%). The majority of respondents were qualified with
PhD or higher (60%), and just over half were appointed to a Level C (Senior Lecturer)
position or higher (56%). (Some did not provide demographic details.)
Data management and analysis
Responses to the open-ended questions were entered were coded:
 to identify which of the eight researcher attributes was being described, and
 to interpret and categorise the descriptions given.
Categories developed to capture descriptions given for each of the attributes
included items such as personal characteristics (e.g., committed, enthusiasm, humility,
good organisation); intellectual factors (e.g., enquiring/open mind, creative/innovative);
having research expertise (e.g., methodological understanding, analytic skills,
substantive knowledge, good communicator); the nature of their research (e.g., breadth,
niche, impact, depth, trendy); and a range of other interpersonal and behavioural
attributes (e.g., builds networks, neglects other duties, collegial, peer esteem,
disseminates); background or contextual factors (e.g., education and training,
opportunity, resources); areas of relevance for research (e.g., social relevance, industry
relevance, academic relevance); and outcomes of performance (unelaborated references
to such things as publications, awards, citations).
Use of computer software (NVivo) ensured that full text for each code remained
accessible, and allowed for adjustments to the coding scheme as developments occurred
in thinking about the categories being used. For example, some of the original
categories were combined along the way as they appeared to be covering much the same
thing (e.g., creativity, innovation and originality), and others were divided in order to
refine the description (e.g., focused and niche were differentiated because the former
related more to the way the researcher approached their work, while the latter focused
on the nature of the research being undertaken). All coding categories were reviewed at
the completion of coding to ensure consistency in what was coded (Boyatzis, 1998), and
to create a summary description of each code used. The descriptor codes were then
cross-tabulated against the eight researcher attributes that had been used as prompts,
using the matrix coding query function in NVivo, with the cells in the table showing the
number of times each descriptor was used for each of the eight researcher attributes and
providing immediate access to relevant text (Figure 1). Some later refinements were
made to the coding system during the course of preliminary analyses and concept
development, as described below (Results 2)—categories and numbers reported in
Results 1 were then recalculated to reflect those changes.
Figure 1. Cross-tabulation of researcher attributes with detailed descriptors for research activity
Results 1: Descriptions of high-level research and researchers
The pressure to publish (linked with publication as an outcome), not surprisingly, dominates
researchers’ thinking about research performance. The next cluster of descriptors for
performing as a high-level researcher includes expression of creativity, originality and
innovation, while drawing on sound analytic and intellectual skills in producing new, sound,
and/or relevant knowledge for the academy (the latter being sometimes but not always linked
with publication and citations). These are followed by commitment, peer esteem, social-
practical relevance and methodological soundness. Sharing ideas and skills with others was
used frequently (second, on the basis of raw counts but not when weighted for importance),
but almost entirely for describing an approachable researcher, and approachability was rated as
being relatively unimportant for performance as a researcher (compared to the other seven
attributes) by this sample of academics.
Drawing on the cross-tabulation of researcher attributes with descriptions, the primary
categories used to describe each of the eight researcher attributes were identified. These are
shown in Table 1, with exemplars of how they were expressed. Each type of researcher (i.e.,
each attribute) was uniquely described, although the possibility is there for some clustering
based on relative similarity in the pattern of descriptors used (e.g., productivity with
recognition).

Table 1. Characterising eight researcher attributes

Primary descriptors used
Quality
 knowledge for the academy
 publishes
 creative/innovative
 analytic-intellectual skills
 methodologically sound
 peer esteem
Ability
 analytic-intellectual skills
 creative/innovative
 commitment/persistence
 methodologically sound
 substantive knowledge
 organised/disciplined
Productivity
 publishes (leading to
publications)
 volume of work
 knowledge for the academy
 finisher
N* Exemplar descriptions
82 Very creative…uses new approaches or shows insight to link
different areas together.
67
Innovative and of relevance to discussion within discipline;
63 thoroughly carried out.
52 Original, thorough and meticulous work, well-argued, well-
48 read, well-researched discussion.
44 Ability of work to undergo peer review; ability of work to be
read/understood by wider audiences...
110 Able to analyse problems and identify the significant issues
that need to be addressed - can prioritize elements of the
83
problem.
60 Ability is usually a combination of brute intelligence, firm
54 analytic skills, motivation, discipline and a gift for finding
49 good problems.
47 Imaginative conceptualisation. Well read in their areas.
Good at organising their time and fairly ruthless in guarding
it against competing demands.
167 Someone who brings research projects to fruition - resulting
in publications.
Regular presenter at scientific conferences, published a great
63 number of papers in respected, refereed journals.
38 Number of papers published, number of research associates
30 and research students in research group, number of grants.
Recognition
 publishes (leading to 115 They publish, they present lectures, their work is referred to
and respected by others.
publications)
Self-promotion. Good at identifying and targeting
 peer esteem 54
prestigious outlets, including electronic media.
 inclined to self promotion 49 Has the knack (as well as much talent) of being invited all
 knowledge for the academy 45 over the world to lecture etc.
Benefit
 social-practical relevance 103 The development of ideas which give real insight to the
human condition [or] solution of practical problems.
 knowledge for the academy 79
Meaningful contributions to society; being able to
 industry relevance 60 communicate to people in ways that help to make the benefits
realisable.
Contributions to professional practice, productivity,
contributions to theory are all valuable and necessary.
Benefit can only be measured on a fairly long time scale.
Active
 commitment/persistence 69 They do research rather than talk about it.
Participate in the process of research themselves.
 publishes 62
Active researchers these days have to be very energetic,
 hands-on researcher 54
forceful and bloody-minded if they are to succeed.
Those that are always missing from staff meetings.
Satisfaction
 intrinsic interest 67 Enthusiasm for their field; great curiosity and a love of
problem solving.
 enthusiasm 45
…genuine interest in the problems they investigate.
 peer esteem 45
The satisfaction for some comes from public recognition; for
others, from their own regard for what they do. Others again
simply love the process of research as a way of life.
Some gain satisfaction from their influence, power and status.
Approachability
 shares ideas, listens, helpful 161 Willingness to share ideas and happy to discuss those (ideas)
of others.
 explains to others 70
Capable of explaining a complex piece of work at a level
 enthusiasm 21 easily understood.
 humility 21 Confident in their own ability and knowledge. Not defensive
and secretive with know-how and information.
They are eager to test their theories and enjoy the cut and
thrust of critical debate. They are not 'precious' about their
research.

* Number of people using the listed descriptor for the identified researcher attribute
 Overall, quality and ability were seen to be most evident in creativity, innovation
and originality supported by a high level of research skills and personal application.
Productivity and recognition, in contrast, are evidenced more by dissemination of work,
especially through publications, with recognition being aided also by networking and
making oneself known. These four attributes attracted fuller descriptions than the
remaining four, and the nature of the descriptions given reinforces the notion of a
significant quality-quantity divide in observations of research performance, with
attributes such as benefit, satisfaction and approachability being independent of this
dichotomy.

Results 2: Dimensions and structure of research performance

Concepts are typically multidimensional and multilevel (Goertz, 2006). At the basic
level, a concept is a noun to which adjectives might be added. At the secondary level,
dimensions are described—these are more often ontological rather than causal in their
relationship with the basic level concept. The third level is typically an indicator level.
Based on the descriptions of high-performing researchers given by 295
academics from across a wide range of disciplines, representing different levels of
qualification and appointment and from universities with varied traditions, a conceptual
model of research performance is proposed. Sixty descriptors, simply arranged
alphabetically during coding, were reviewed and sorted by placing them as items in a
visual model. They were progressively arranged and rearranged on the screen, and
labels were attached to clusters. Further review involved reshaping descriptor codes and
redefining cluster labels as the dimensions of the concept were clarified; this continued
through the initial writing process, often stimulated by having to justify a descriptor’s
presence in any cluster. Text coded for any descriptor was able to be retrieved, reviewed
and recoded as needed throughout this process. Where codes were considered for
combination, they were reviewed in at least two ways before merging: firstly, the text
for each was temporarily merged for review, with marginal coding stripes showing their
source codes, before final merging; secondly, similarity in the pattern of distribution of
those codes across the eight attributes used as prompt questions was assessed using both
numbers and text. The model was ‘juggled’ and reworked until all descriptors provided
by respondents could be meaningfully accounted for. The final conceptual model
comprised a reduced set of 40 descriptor codes, with 29 serving as components (and
potential indicators) of the core dimensions of research performance (Figure 2).
Six dimensions constitute research performance, for which there are additionally
two necessary pre-conditions and three consequential outcomes (although causally
associated with it, these pre-conditions and outcomes do not form part of the actual
model of research performance). Four of the six constitutive dimensions relate to
conducting research and two to making the research being undertaken visible to others.
In developing her ‘thick vague conception [of] the shape of the human form of life,’
Nussbaum noted: ‘The list is an intuitive approximation, whose purpose is not to cut off
discussion but to direct attention to certain features of importance’ (1992, p.216).
Similarly, this is unlikely to be the final word on a concept of research performance, but
hands-on; intrinsic interest; enthusiasm; engagement
breadth; volume; neglect other duties

*
organised-disciplined; careful-thorough; task
finisher; commitment-persistence; orientation
problem solver; confident
Research
* activity

methodologically sound; technical skill; research

substantive knowledge; strategic practice

*
creative-innovative; analytic thinker; intellectual Research
curiosity-open mind process
* performance

publishes; presents; communicates well; dissemination

uses networks; self-promoting
Performing –
+ making visible
explains to others; shares ideas-helpful; collegial
research leader; humility; has students engagement

Legend:
ontological dimension
substitutable
conjunction of non-causal necessary conditions
+ logical OR
* logical AND

Figure 2. Conceptual model of the dimensions of research performance

is proposed in order to emphasise oft-neglected features and promote discussion and
further research.
Basic-level phenomenon of research performance
The term ‘research performance,’ itself, implies two basic components: ‘research’ –
there must be activity of some sort that can be considered to be research, as a basis for
performance; and ‘performance’ – what is learned through the research must be made
visible and passed on (performed) to others. Both of these components were referenced
in the descriptors provided by the academic researchers in this study.
Secondary-level dimensions of research performance
Engagement (interest and involvement). Research activity is characterised by
enthusiastic, preferably ‘hands-on’ involvement. Intrinsic interest in the topic and/or a
breadth of interest drive the engaged researcher, at times to the point of their neglecting
other responsibilities. Without engagement, research simply does not occur. Some
engage in research by proxy (as when a professor works through assistants and
students)—whether this is satisfactory in creating high level performance depends
largely on the degree of direct involvement (engagement) as well as the leadership and
mentoring skills of the chief investigator. Engagement at some level is an essential
component of research performance.
Task orientation (disciplined management—getting the job done). The task of research
is approached with commitment and persistence, to the point of completion. The
researcher solves problems and works confidently, in an organised and disciplined
manner and with care and thoroughness. The standard of performance is conditional on
the degree to which researchers demonstrate these attributes.
Research practice (knowledge and skill—substantively and methodologically sound).
The researcher has a deep understanding of the substantive topic being researched. The
substance of the research is also strategic: for example, researchers ‘recognise and
define a problem worthy of research’ and focus on that. Excellent researchers’ work is
methodologically appropriate and technically skilful. Without knowledge and skill, the
research will be trivial and/or open to significant criticism, and will be dismissed.
Intellectual processes (analytic capacity and creative thinking). Research is essentially
an intellectual activity. It requires a high level of interpretive and analytic capacity in
the researcher, and a mind open to new and different ways of seeing things. Excellent
research is defined by originality; it is characterised by creativity and innovation. At
times it involves lateral thinking and flexibility to adapt and move in new directions as
evidence (and sometimes intuition) directs. Rigidity, lack of imagination, and inability
to deconstruct and reconstruct a problem will not generate interesting, ground-breaking
research, if it generates any at all.
Dissemination (formal communication of research outcomes). Dissemination ensures
that the research being undertaken is brought into the public arena: it is made visible so
that others can benefit from it. While publishing is seen as the primary means of
dissemination, with the status of the journal often being noted, other avenues of
communication, such as conferences and seminars, are also important. Dissemination,
particularly in these latter forms, is assisted to some degree by a capacity for self
promotion and an ability to build and use networks amongst peers. Lack of
dissemination in these forms is a severe constraint on performance, but high-level
performance is not directly related to the sheer volume of dissemination.
Collegial engagement (sharing knowledge and expertise). Sharing one’s expertise
(substantive or methodological) with others collegially, or in a leadership or supervisory
capacity, is an additional (and for some, an alternative) way of passing on research
knowledge. In a collegial or leadership role, a quality of humility is appreciated. This is
facilitated by a kind of confidence that leads to being able to share without feeling
‘precious’ about ‘giving away’ ideas or having one’s ideas critiqued. Collegial
performance contributes to the work of the whole team or centre or department, and
ensures a future for new generations of researchers. Additionally, it has the benefit of
feeding back into improving the activity and performance level of the performer as they
learn, reciprocally, from those around them. Without this dimension of collegiality high
level performance can exist, but it takes on a particular self-serving quality that may be
in danger of losing relevance.
Conditions for research performance
Most respondents did not address this issue, but those who did pointed to two further
dimensions necessary for research performance to occur. Training and experience, as
part of the background of the researcher, provide an essential foundation for skills
required in order to perform at a high level. Opportunity and resources, including time,
equipment and funding, as part of a supportive institutional environment, are also
necessary (but not sufficient) conditions which need to be met if research of any
significance is to occur.
Outcomes of research performance
Performance results in three broad classes of outcomes. The most obvious of these is
that there is a product which results from performance—most commonly seen as being
in the form of some kind of publication such as a journal article, book or report.
Secondly, research may have an impact on others’ (academic) research or knowledge,
on industry, in practical ways in society, or through changing the way we think about
ourselves as human beings. Thirdly (and finally), research performance enhances the
reputation of the researcher, creating peer esteem and potentially leading to invitations,
awards, and promotion, as well as influencing the likelihood of further funding.
Dimensional structure
To adequately describe a concept requires more than simply listing the dimensions, or
even the dimensions and indicators. It is necessary also to define the structure of the
concept—whether and how many of the conditions are ‘necessary and sufficient,’ or
whether a ‘family resemblance’ model (in which any items making up a specified-size
subset would be sufficient) is being proposed (Goertz, 2006). Additionally, if
dimensions are going to lend themselves to measurement, they need to be directional,
with both poles specified (as is the case above).
 All four research activity dimensions—engagement, task orientation, research
practice and intellectual processes—are necessary (and sufficient) to create new
knowledge for passing on to others. They are seen as being ontological rather than
causal, that is, these four dimensions comprise (are what makes for) research activity,
and without any of them, you would have something less than research activity insofar
as it is a component of academic research performance. Ideally, both formal
dissemination and collegial engagement are present in the ‘passing-it-on’ component of
research performance, but one or other may be sufficient. Again, neither of these causes
performance, they constitute performance. Thus, research performance comprises five
necessary dimensions, four of which are non-negotiable. Each dimension is a
continuum, with positive and negative cases defined, rather than dichotomous, just as
there are degrees of performance overall.
Positive cases of research performance can be defined by the existence of
positive values on the four activity dimensions plus a high value on at least one of the
performance dimensions. Negative cases of research performance can take at least two
different forms: those who are actively researching but who do not pass it on in any
way; and those who are engaged in some way (and possibly passing on what they are
doing), but who do not exhibit positive values on any or all of the remaining three
activity dimensions. Irrelevant cases are those who are simply not engaged in research
activity, which is sufficient to make the other dimensions non-applicable. Each of these
alternative non-positive forms of research performance has different causal
implications, for example, for programs offering support or development, as well as for
assessment of performance.
Adding the adjective ‘valued’ to ‘research performance’ increases the intension
(specificity) of the concept, suggesting that indicators for all dimensions should be
placed at or close to the positive pole—something similar, perhaps, to the idea behind
5* in the British research assessment system. The inverse of intension is extension,
where necessary conditions are dropped from the model in order to increase the number
of cases it might include (Goertz, 2006). In this case, to extend the concept one might
specify only that engagement and dissemination are necessary conditions, thereby
including the kind of researcher that I have specified above as a second form of negative
case, evidenced perhaps in poor presentation based on superficial analysis of inadequate
data. Extension of the concept, therefore, is not recommended.

Discussion
Data from a cross-section of academics across a variety of universities have been
analysed to explore academic perceptions of the characteristics of high-performing
researchers and to build a dimensional, conceptual model of research performance. The
dimensions constituting research activity and performing identified through the
interpretive conceptual analysis of the qualitative data provide a basis for challenging,
extending and re-building outcome-focused models currently driving research
assessment exercises. One can argue, for example, that engagement, task orientation,
research skills and intellectual processes will be most evident in the quality (rather than
quantity) of output from research, and hence these dimensions and their indicators can
be used to give more specific meaning to that term. Furthermore, the dimensions and
indicators identified in this study could have particular relevance at the local and
individual level of assessment, when the ‘research active’ status of an academic,
department or centre is being determined for workload or other purposes. This is
something that is mismanaged and discriminatory in the current system in Australia, if
not elsewhere (Bazeley, 2006a).
To the extent that the concept of research performance that has been developed
in this study is based on empirical data, the items included have been impacted by the
questions asked and the form of asking, and perhaps also by Australian academic
culture of the mid-1990s when the data were collected. The danger inherent in brief,
self-report questionnaires is that respondents will ‘dash off’ answers without deep
reflection, and this is apparent here, for example, in the relative level of emphasis on
simply ‘having publications’ as a desirable characteristic of researchers. This contrasts,
to some extent, with Åkerlind’s (2008) finding of an equivalent emphasis being given to
personal and real-world benefits stemming from research, and with data gathered in a
non-assessment context which found that professional and public audiences were at
least as important, if not more, than academic audiences for researchers working in
social science, professional and humanities disciplines (Bazeley, 2006a).
That the empirical data was based on a cross-disciplinary sample of academics
with different levels of personal research expertise and from established, middle-
ranking and new universities contributes to the potential universality of the concept.
What the researcher brings to the data analysis and conceptualisation, to intuitively add
to the breadth of that perspective, is almost 40 years of research experience, with 20 of
those working in a cross-disciplinary and cross-national developmental role. Thus, the
concept of research performance is intended to be universal at the dimensional level. At
the indicator level, however, those in different disciplines (and at different levels of
maturity) will quite possibly require different emphases to reflect the dimension for
their discipline (Åkerlind, 2008). This issue will be explored in a further article, along
with the possible influence of gender and qualification (as a proxy for experience).
A number of interesting dilemmas were faced in building and proposing this
conceptual model. The equation between volume of published output and impact on the
research community has long been a matter of contention. I have proposed that products
of research (such as publications of various kinds), impact and indicators of enhanced
reputation should be viewed as consequences of performance—yet these three things
are what most measures of performance attempt to assess. When Steele (2004. p. 67)
suggested that publication had become more important than dissemination, he went on
to say that the results of research have ‘often been disseminated well before the
publication. The publication is for the accreditation and tenure.’ While the act of
publishing is a form of dissemination (unlike citations, invitations or awards), and was
seen as such in the process of coding the data, it is important, nevertheless, to
distinguish conceptually between the action and its outcome, as this model is designed
to do.
Perhaps my most controversial decision was to view collegial engagement as an
alternative dimension of the performing component of research performance. The
reason for doing so is well presented in a description given of an active researcher by a
female senior scientist: ‘Full of ideas, willing to communicate those ideas – often at the
expense of conventional research articles, sharing ideas with students or colleagues is a
higher priority.’ One is reminded that much of George Herbert Mead’s seminal work
has been published by and through his students, so that the impact he has had on social
psychology and related social sciences has been as much, if not more, though that
mentoring/teaching/inspiring route than through directly publishing. Collegial
communities of practice and micro communities of knowledge, with their vital
interpersonal communication channels, are a primary means of converting tacit
knowledge, generated through knowledge creation projects, into explicit knowledge—
yet these are being eroded in the new competitive and isolating environment of
universities (Moss & Kubacki, 2007).
The necessity for ‘hands-on’ engagement points to one further issue in what
makes for research performance. Many researchers regarded as highly productive
‘spend little time doing research themselves; [they] focus on money to employ people
and getting names on publications’ (male, senior scientist). Apart from the significant
number of respondents who mentioned active personal involvement as being important,
some years ago Frost and Stablein (1992) pointed to the significance of ‘handling your
own rat’ for excellence in research. The professor who finds that their assistant or junior
colleague, who was responsible for carrying out a major part of the research process,
has disappeared shortly before a report is due can be placed in a very awkward
situation! Desirable qualities for a researcher, such as being intensively engaged in
one’s research (like any obsession), can have negative side effects, however, such as the
complaint registered in this data that such people are often neglectful of their other
duties (they are often neglectful of their families as well).
Further substantive analyses of this data will identify whether interpretations of
the dimensions in the model vary in relation to discipline, gender, and experience, as
noted above. To empirically test the necessity for each of the dimensions and to refine
the indicators outlined above, further data will need to be gathered. Intensive case
studies of a purposive sample of academics at different points in their research career
would provide a useful first assessment, would assist also in seeing more clearly how
desirable behaviours might be evidenced in work done, and would help to clarify
potential issues around the nature and extent of the role played by the published paper in
effective academic communication and advancement of research. Measures based on a
refined set of indicators might then be developed and tested in discriminant analyses
with groups selected to vary by broad consensus in their level of research performance.

Acknowledgements
Initial work on this study was supported by the Australian Research Council. Further analytic work was
undertaken while I was Visiting International Fellow at the Institute for Social Research at the University
of Surrey in 2006.
References
Åkerlind, G. S. (2008). An academic perspective on research and being a researcher: an integration of the
literature. Studies in Higher Education, 33(1), 17-31.
Archer, L. (2008). Younger academics’ constructions of ‘authenticity’, ‘success’ and professional
identity. Studies in Higher Education, 33(4), 385-403.
Batterham, R. (2004). Measuring excellence: A chief scientist perspective. In National Academies Forum
(22 June), Measuring excellence in research and research training, pp.3-8. Canberra: The
Academy of Science.
Bazeley, P. (2006a). Research dissemination in Creative Arts, Humanities and the Social Sciences.
Higher Education Research and Development, 25(3), 215-229.
Bazeley, P. (2006b). The contribution of computer software to integrating qualitative and quantitative
data and analyses. Research in the Schools, 13(1), 63-73.
Bazeley, P. (2009a). Mixed methods data analysis. In S. Andrew & E. Halcomb (Eds.), Mixed methods
research for nursing and the health sciences (pp. 84-118). Chichester, UK: Wiley-Blackwell.
Bazeley, P. (2009b). Integrating analyses in mixed methods research [Editorial]. Journal of Mixed
Methods Research, 3(3), 203-207.
Brew, A. (2001). Conceptions of research: a phenomenographic study. Studies in Higher Education, 26,
271-285.
Bourdieu, P. (2001). Homo academicus. Cambridge: Polity Press.
Bowden, J., Green, P., Cherry, N., & Usher, R. (2005). Academics’ ways of understanding success in
research activities. In J. Bowden & P. Green (Eds.), Doing developmental phenomenography
(pp. 128-144). Melbourne: RMIT University Press.
Boyatzis, R. E. (1998). Transforming qualitative information: thematic analysis and code development.
Thousand Oaks, CA: Sage.
Bruce, C., Pham, B., & Stoodley, I. (2004). Constituting the significance and value of research: views
from information technology academics and industry professionals. Studies in Higher Education,
29, 219-238.
Creamer, E. G. (1998). Assessing faculty publication productivity: issues of equity (Vol. 26). Washington
D.C.: The George Washington University, Graduate School of Education and Human
Development.
Frost, P. J., & Stablein, R. E. (1992). Doing exemplary research. Newbury Park, CA: Sage.
Goertz, G. (2006). Social science concepts: a user's guide. Princeton, NJ: Princeton University Press.
Marton, F. (1981). Phenomenography - describing conceptions of the world around us. Instructional
Science, 10, 177-200.
McNay, I. (2003). Assessing the assessment: an analysis of the UK Research Assessment Exercise, 2001,
and its outcomes, with special reference to research in education. Science and Public Policy,
30(1), 47-54.
Moss, G., & Kubacki, K. (2007). Researchers in higher education: a neglected focus of study? Journal of
Further and Higher Education, 31(3), 297-310.
Nussbaum, M. C. (1992). Human functioning and social justice: in defence of Aristotelian essentialism.
Political Theory, 20(2), 202-246.
Steele, C. (2004). Changing research practices in the digital information and communication
environment. In National Academies Forum (22 June), Measuring excellence in research and
research training, pp. 61-71. Canberra: The Academy of Science.
Steele, C., Butler, L., & Kingsley, D. (2006). The publishing imperative: the pervasive influence of
publication metrics. Learned Publishing, 19, 277-290.
Tight, M. (2004). Research into higher education: an a-theoretical community of practice? Higher
Education Research and Development, 23(4), 395-411.