Higher Education Quarterly, 0951–5224

DOI: 10.1111/j.1468-2273.2011.00499.x
Volume 66, No. 1, January 2012, pp 47–64

The Impact of Research

Training and Research Codes
of Practice on Submission of
Doctoral Degrees: An
Exploratory Cohort Study hequ_499 47..64

Robin Humphrey, Newcastle University,

robin.humphrey@ncl.ac.uk
Neill Marshall, Newcastle University, j.n.marshall@ncl.ac.uk
Laura Leonardo, Newcastle University,
laura.leonardo@ncl.ac.uk

Abstract
The paper examines the impact of the transformations in doctoral education in
the arts, humanities and social sciences in the United Kingdom over the past
decade. It focuses on the introduction of formal research training and codes of
research practice and in the first longitudinal candidate cohort study examines
their impact on doctoral outcomes, especially Ph.D. submission rates. Results
from this quantitative study show that engagement with research training,
completion of a project outline and plan and appointment of a supervisory team
were statistically positively associated with submission of the thesis within four
years. It is concluded that the professionalisation of doctoral education by
research training and codes of research practice has had a positive impact on
doctoral educational outcomes.

Introduction
This paper examines the impact of the introduction of formal research
training and codes of research practice on the education of doctoral
candidates in one UK higher education institution. In the first longitu-
dinal candidate cohort study it focuses on the impact on one of the key
doctoral outcomes, submission rates, in the disciplines where these rates
historically are at their lowest, in the humanities and social sciences
(Council of Graduate Schools, 2008).

© 2011 The Authors. Higher Education Quarterly © 2011 Blackwell Publishing Ltd, 9600
Garsington Road, Oxford, OX4, 2DQ, UK and 350 Main Street, Malden, MA 02148, USA.
48 Higher Education Quarterly

Ever since the expansion in postgraduate study in the UK following

publication of the Robbins Report in 1963, there has been concern over
the length of time some Ph.D. candidates take to submit their theses. A
succession of UK studies have found academic discipline to be the most
reliable predictor of successful submission, with doctoral candidates in
the sciences having persistently higher rates of submission than those in
the humanities and social sciences (Welsh, 1980; Bowen and Ruden-
stine, 1992; Booth and Satchell, 1995; Wright and Cochrane, 2000). In
the context of a rapid global expansion of doctoral education (Nerad,
2006), these concerns have been echoed in Australia (Sinclair, 2004) and
the USA (Council of Graduate Schools 2008, 2009, 2010).
There are several reasons for this protracted period of Ph.D. produc-
tion in the humanities and social sciences, including limited state and
institutional funding for full-time Ph.D. research, and the associated
vagaries of self-funding, and the extensive nature of the Ph.D. thesis
itself, which addresses multiple goals including providing a measure of
research excellence, an entry qualification to the academic profession
and an indicator of the assimilation of professional research practice. An
additional factor in the humanities and social sciences is that most Ph.D.
candidates design their projects themselves, rather than work on one
already created by their supervisor as in the natural sciences. For the
former, much of the initial period following registration is spent refining
ideas and tightening research design, almost inevitably lengthening the
time to submission in the process.
Doctoral research education and training have, over the last two
decades, changed profoundly in Europe (Kehm, 2005, 2007, 2010),
Australasia (Pearson, 2005; Kiley and Mullins, 2006; Pearson et al., 2008)
and the USA (Goldberger et al., 1995; Nyquist and Woodford 2000;
Nerad and Heggelund, 2008). The Bologna Process in Europe (Joint
Declaration of the European Ministers of Education, 1999; EUA, 2007)
and national governments in the UK (QAA, 2004) and Australia (West,
1998; Kemp, 1999a, 1999b), along with a combination of federal govern-
ment and national foundations and organisations in the USA (Austin
et al., 2009), have all been important influences on the nature and extent
of doctoral education globally. Codes of practice have been introduced
covering admissions criteria and the responsibilities of supervisors and
students. Regulations and policies have been developed to direct doctoral
education. This has become more structured and systematic and no
longer the preserve of an individual academic expert working with a
student apprentice. This increased steering of doctoral education and
training reflects a belief in the important contribution of research and

© 2011 The Authors. Higher Education Quarterly © 2011 Blackwell Publishing Ltd.
The Impact on Submission of Doctoral Degrees 49

development to national income and competitiveness in an increasingly

knowledge-intensive economy (McWilliam and James, 2002; EUA, 2005;
Davis et al., 2006; Nerad, 2006). In the context of a growing demand for
doctoral education, and increasing international mobility of students and
academic staff, governments and institutions also wish to ensure value for
money for their investment in doctoral education. There is also a wider
appreciation of the importance of higher-level skills and qualifications for
research careers outside academia.
In the UK, this professionalisation of the doctorate has resulted in
three major national developments. First, the identification of best prac-
tice in doctoral education (Harris, 1996; Tinkler and Jackson, 2000,
2002, 2004) has led many institutions to introduce new measures and to
tighten their procedures; and this process was formalised with the pub-
lication of the Code of Practice for Research Degrees by the Quality
Assurance Agency for Higher Education (2004). This code introduced
minimum standards for research degree programmes, including sti-
pulations about rigorous external examining to ensure standards were
maintained. Alignment with the Code has, since 2006–07, been audited
by the Quality Assurance Agency within its regular programme of insti-
tutional audits.
Second, in response to government scrutiny, which identified poor
completion rates for their funded studentships (Advisory Board for the
Research Councils, 1982; Winfield Report, 1987), UK research councils
introduced research training requirements and submission rate monitor-
ing to increase the likelihood that doctoral candidates in receipt of
studentships submitted within four years. The Economic and Social
Research Council (ESRC) was in the vanguard of this process and
introduced extensive discipline-specific and generic research training
requirements in 1994 (subsequently updated in 2006 and 2009) as a
prerequisite for the recognition of institutions eligible to receive its
studentships. At least 60% of ESRC-funded doctoral candidates from a
given institution had to submit in four years for it to remain eligible for
studentships. Institutions that returned a lower submission rate would be
ineligible to receive new ESRC research studentships for up to two years.
The Arts and Humanities Research Council followed the ESRC’s lead
and, in 2009, increased its institutional submission target to 70%. Both
research councils claim that the operation of this scrutiny policy has
dramatically improved overall submission rates; in the ESRC’s case from
less than 30% in the early 1980s to the current level of 79% (ESRC,
2010). In recent years, the Higher Education Funding Council for
England (2007, 2009) has also monitored Ph.D. qualification rates,

© 2011 The Authors. Higher Education Quarterly © 2011 Blackwell Publishing Ltd.
50 Higher Education Quarterly

producing league tables comparing institutions via a benchmark reflect-

ing their mix of disciplines and number of research council students.
The third major driver for change in the UK was the introduction of
transferable skills training and development for doctoral candidates fol-
lowing the publication of the influential Roberts Report (Roberts, 2001),
which resulted in additional ‘ring-fenced’ government funds being made
available for research-council-funded students. The requirements for
transferable skills development were set out in the UK’s Joint Statement
of the Research Councils’ Skills Training Requirements for Research
Students (RCUK, 2002). The skills were grouped into seven categories
covering: research skills and techniques; research environment; research
management; personal effectiveness; communication skills; networking
and team working; and career management. Although Roberts funding
ends in 2010–11, research training and transferable skills development
are now embedded within the structures of higher educational institu-
tions and will continue to be delivered and developed in response to the
publication by Vitae, a national organisation funded through Roberts, of
the new Researcher Development Statement (Vitae, 2010), which will
replace the Joint Skills Statement.
Doctoral education has received considerably less research attention
than other aspects of higher education. However, stimulated by the
Higher Education Funding Council for England’s annual monitoring of
Roberts expenditure, an exploration has taken place of the impact of the
Roberts initiative.To encourage higher education institutions to evaluate
their training and development, the Rugby Team, a sector-wide group
specialising in doctoral education, has developed an impact framework
(Bromley et al., 2008), which identifies five impact levels for researcher
training and development. These are: foundations (measures inputs and
throughput relating to infrastructural investment in staff, programmes
and facilities); reaction (feedback from participants); learning (attitudi-
nal changes, improved knowledge, increased skills); behaviour (behav-
iour changes); and outcomes (better research, timely submission, more
employable candidates).
In a sector-wide report by Bromley (2010) for the Impact and Evalu-
ation Group (formerly the Rugby Team), data has been collated from
120 studies, using attendance and feedback as the main performance
measures.This shows that, according to the Rugby Team Impact Frame-
work evaluation model (Bromley et al., 2008), there is substantial evi-
dence to suggest that researcher development initiatives have positive
impacts on four of the impact levels (foundations, reaction, learning and
behaviour) of the impact framework. A recent review of progress in

© 2011 The Authors. Higher Education Quarterly © 2011 Blackwell Publishing Ltd.
The Impact on Submission of Doctoral Degrees 51

implementing the Roberts recommendations commissioned by Research

Councils UK (RCUK, 2010) concluded that, according to qualitative
evidence, major improvements had been enabled by so-called Roberts
money. However, the review also concluded that it was impossible to
quantify the impact of Roberts money, largely because no firm baseline
had been established at the outset.
The complexity of the relationship between research training inputs
and doctoral outcomes has made evaluation difficult. Doctoral outcomes
are various and include measures of employability, educational attain-
ment and the wider student experience, which in the UK has been
captured by the Postgraduate Research Experience Survey conducted by
the Higher Education Academy. Training is only one of a number of
influences on doctoral outcomes; a range of personal, financial and
discipline-specific factors, including the character of supervision, have
also been found to be important (Cryer, 2000; Golde and Dore, 2004;
Council of Graduate Schools, 2010).
The study reported in this paper is the first longitudinal cohort
study to assess the impact of research training and development and of
code of practice measures, through use of the key outcome measure of
thesis submission, chosen because of the interest of policymakers and
Research Councils in the UK in this outcome measure. This is the first
quantitative evaluation of impact on outcomes of the Rugby Team
Impact Framework.

The Newcastle University context

This paper is based on an analysis of data from one large faculty in one
institution. Newcastle University is a research-led institution and a
member of the Russell Group, an association of the 20 major research-
intensive universities in the UK. Newcastle is one of the UK’s largest
civic universities, with over 18,000, mostly full-time, students, includ-
ing over 4,000 postgraduates. The Faculty of Humanities and Social
Sciences (HaSS) is the largest of the three faculties of the university
and covers a very wide range of disciplines, including fine art, eco-
nomics, architecture, music, education, modern languages, politics,
business, planning, English language and literature, sociology, history,
archaeology and geography. The faculty has a dedicated graduate
school with a purpose-built postgraduate training suite and, in 2009–
10, 792 students were registered on doctoral programmes. The Gradu-
ate School monitors the implementation of the measures introduced by
the university’s ‘Code of Practice’, introduced in 2004–05, which was
modelled on the Quality Assurance Agency for Higher Education’s

© 2011 The Authors. Higher Education Quarterly © 2011 Blackwell Publishing Ltd.
52 Higher Education Quarterly

national code. Key components include a learning agreement com-

pleted at the outset of their studies by students and their supervisors,
project approval, which includes a project outline and plan written by
the student, and an annual review of student progress. The faculty’s
postgraduate research training and researcher development programme
is provided to all doctoral postgraduates registered in the faculty. This
was initiated in response to the ESRC’s postgraduate research training
guidelines in 1994 but has subsequently expanded its remit to include
provision for arts and humanities doctoral candidates and Roberts
transferable skills. A recent audit concluded that the postgraduate
research training and researcher development programme met all the
requirements of the Joint Statement of the Research Councils’ Skills
Training Requirements for Research Students outlined above. A key
introductory module, ‘Managing a Ph.D.’, includes sessions on: plan-
ning, organising and managing a Ph.D.; good academic conduct;
ethical questions in research; writing literature reviews; and managing
relationships with supervisors. The completion by students of an
assessment for this module provides one of the variables included in
the data set for this study.

The data set

The study is based on a cohort study of those Ph.D. students first
registered in 2004, the first cohort to start their period of registration
after the introduction of the university’s code of practice and core
research training requirements were introduced and to have been regis-
tered long enough to be expected to submit their thesis within the four
years, barring interruptions. During the early stages in the introduction
of research training and the code of practice, compliance for some
procedures was uneven across the Ph.D. cohort.This allows comparisons
to be made between those who engaged with the new procedures and
those who did not. The study was made possible by collating the
detailed, longitudinal data on this cohort kept by the faculty’s Graduate
School. Routine data concerning each candidate’s registration were
compiled, including: start date; supervisory team; date of thesis submis-
sion; viva outcome; date of completion (pass list). Data were then added
on compliance with code of practice procedures, namely submission of
a learning agreement between the candidate and supervisor(s) and a
project approval form (PAF). Finally, data on engagement with the
postgraduate research training and researcher development programme
were added, as measured by submission of a 1,500-word assessment for
the ‘Managing a Ph.D.’ module. All except five of the doctoral candidates

© 2011 The Authors. Higher Education Quarterly © 2011 Blackwell Publishing Ltd.
The Impact on Submission of Doctoral Degrees 53

had submitted a learning agreement, some in response to a drive by the

Graduate School to raise completion rates. Therefore, this factor, which
does not discriminate between students, will not be used in the subse-
quent analysis.
Eighty-seven full-time research Ph.D. candidates were registered in
HaSS in 2004. Part-timers were excluded from the study because insuf-
ficient time had elapsed for them to submit by 2009 when the data were
assembled. International students (those whose country of origin was
outside the EU) accounted for 58 (66%) of the cohort and there were
slightly more men (46: 53%) than women. There were more doctoral
candidates in the social sciences (49: 56%) than in the humanities,
although all nine schools in the faculty had doctoral candidates register-
ing that year. The schools with the most candidates were Geography,
Politics and Sociology and English Literature, Language and Linguistics,
with 23 (26%) each.
Between 2004 and 2009, a substantial minority of students (12: 14%)
withdrew and a further 23 (26%) were granted official interruptions to
their studies of between two and 25 months (mean = 13 months). Of
those students who were still registered in 2009, 43 (65%) had submit-
ted their theses for examination, with a mean average time to submission
of 47.3 months after interruptions had been taken into account. Of the
remaining 32 research candidates who had still to submit, 19 had had
interruptions (the mean average time of which was 12 months) and these
have also been taken into account in the analysis below.

Bivariate analysis
To test the impact on submission of the new elements in the UK doctoral
process, a series of bivariate, inferential statistical tests was employed.
The three key variables in the data set integral to contemporary doctoral
conditions in the UK were employed in the analysis as predictors for
submission of the doctoral thesis in four years (Table 1).
There is a significant positive association at the 5% level between
engaging with research training and submitting the thesis within four
years, with 63% of candidates who had engaged with research training
submitting within four years, as against only 40% of those who did not.
The significance of this result is not affected by excluding the 12 candi-
dates who had withdrawn their Ph.D. registration from the analysis.
Excluding these withdrawals, 71% of the engaged candidates had
submitted as against 48% of the non-engaged (chi-square = 4.01;
p = 0.045).

© 2011 The Authors. Higher Education Quarterly © 2011 Blackwell Publishing Ltd.
54 Higher Education Quarterly

TABLE 1
Predictors of submission in four years
Submission in 4 years Chi- p
square
Yes No
n % n %
Engagement with
research training
Yes 22 63 13 37 4.23 0.04
No 21 40 31 60
Completion of PAF
Yes 39 56 31 44 5.67 0.02
No 4 24 13 76
Supervisory arrangements
One supervisor 7 32 15 68 3.65 0.05
Supervisory team 36 54 29 45

A similar pattern is also found when completion of the PAF is cross-

tabulated with submission of thesis, with a slightly smaller proportion of
candidates (56%) both completing PAFs and submitting their theses
but a stronger probability of association between these two factors
(p = 0.02).The exclusion from the analysis of the 12 candidates who had
withdrawn their registration had an impact, however, since 7 of those 12
candidates had not submitted a PAF, although this reduction makes
further statistical analysis unreliable since the expected counts in two of
the cells in the contingency table are 5 or less.
The first implication of the results is that research management and
careful planning at the start of doctoral registration are closely linked to
timely submission of the thesis and possibly to the retention of Ph.D.
candidates as well. Although an association does not specify causality, it
could be argued that the candidates who engaged with research training
and submitted PAF forms were the ones who already had better man-
agement strategies in place, were more motivated and engaged with
supervision and therefore more likely to submit in a timely manner.
Anonymous student feedback for the ‘Managing a Ph.D.’ module is
consistently positive, however. This typical comment indicates the ben-
efits for doctoral candidates of attending sessions and starting their
doctoral studies in a constructive way:
The module provided me with an insight on how to prepare myself for the
realities on carrying out research and be prepared for the unexpected and how
to avoid pitfalls. I really feel motivated, as I am a slow-starting person.

© 2011 The Authors. Higher Education Quarterly © 2011 Blackwell Publishing Ltd.
The Impact on Submission of Doctoral Degrees 55

Our experience of managing research training programmes for Ph.D.

candidates over the last decade suggests that the most important factor in
doctoral candidates’ engagement and compliance with training and code
of practice measures more widely is the attitude of supervisor(s), rather
than simply the willingness and motivation of the candidates themselves.
The Postgraduate Experience Survey results also show that students
regard high-quality supervision as one of the most important aspects of
their studies (Park, 2009). There is only limited data available on the
supervision of Ph.D. students. Nevertheless, a central element of the
university’s code of practice, was that each doctoral candidate should
have more than one supervisor. Australian research has shown that
doctoral candidates who had ‘active panels’ of supervisors with whom
they could engage experienced greater satisfaction with their arrange-
ments (Cullen et al., 1994) and we are able here to examine the impact on
engagement and submission of team versus sole supervision. Most Ph.D.
candidates starting in 2004 had supervisory teams of either two (39%) or
three (36%) supervisors; however, 25% still retained the traditional sole
supervisor as new supervisory arrangements took time to bed in. What
effects, if any, did the establishment of supervisory teams have on the
doctoral process and outcomes? More candidates with supervisory teams
were engaged with research training (43%) than candidates with sole
supervisors (32%) but this result was not statistically significant (chi-
square = 0.866; p = 0.352). There were, however, two consequences for
candidates who had sole supervision that are worth noting.
There was a less strong, but still statistically significant, association
between supervisory arrangements and submission of the thesis in four
years, with 32% of candidates with sole supervisors submitting as against
54% with supervisory teams. This result is largely explained by the
relatively large proportion of candidates with a sole supervisor with-
drawing from their Ph.D. registration; 6 (27%) had withdrawn, which
amounted to 50% of the doctoral candidates who had withdrawn in the
cohort as a whole.The numbers of candidates who had withdrawn is too
small for reliable statistical analysis but a pattern is certainly emerging of
those candidates who had sole supervision not engaging with elements of
the code of practice or research training.
There was a very strong association between doctoral candidates with
the traditional sole supervisor and a likelihood of failing to submit a PAF;
50% of doctoral candidates with a sole supervisor submitted a PAF, as
against 91% who had supervisory teams (Table 2). One reason for this
could be conservatism among the academics who supervised candidates
on their own, who might have been less enthusiastic about adopting the

© 2011 The Authors. Higher Education Quarterly © 2011 Blackwell Publishing Ltd.
56 Higher Education Quarterly

TABLE 2
Supervisory arrangements by submission of a project approval form
One Supervisory Total
supervisor team
n % n %
Sumission of a PAF
Yes 11 50 59 91 70
No 11 50 6 9 17
Completion of PAF 22 100 65 100 87
Chi-square = 17.38; p = 0.000.

new code of practice procedures than academics who supervised in

teams, and they would also have been less susceptible to peer pressure to
conform to the new procedures.
Two further bivariate analyses produced interesting results concern-
ing the possible effects of engagement with research training. A wide-
spread concern among Ph.D. supervisors and students is that the
introduction of research training in recent years has added to the work
pressure on doctoral candidates and taken valuable time away from the
conduct of the thesis, particularly during the important first year of
the research.The fear is that engagement with research training increases
the time candidates take to submit their theses.There was no evidence in
this study to support such an assertion: an analysis of variance indicated
that there was no significant difference in time taken to submit according
to whether the candidate had engaged with research training (47.1
months) or not (47.6 months) (F = 0.047; p = 0.83).
The last analysis undertaken related to the relationship between
engagement with research training and the outcome of the viva. The
outcomes of the viva at Newcastle for students registered in 2004 are
listed in Table 3. In that cohort, only one candidate failed to satisfy the
examiners in the oral examination and no candidates were recom-
mended for a Master’s degree or were adjudged to have failed altogether.
Engagement with research training had no discernible impact on the
outcome of the viva. The last column in the table shows the outcomes of
PhD examinations in the faculty for the two years when the Ph.D.
candidates in the 2004 cohort were most likely to submit, 2007–09, and
indicates that the pattern of results for this cohort are broadly in line with
the results overall, with 11% more candidates being awarded a Ph.D.
with minor corrections to be submitted within one month and 7% fewer
having to submit their minor corrections within six months.

© 2011 The Authors. Higher Education Quarterly © 2011 Blackwell Publishing Ltd.
 TABLE 3
Submission of thesis by viva outcome for students registered in 2004 and all students submitting 2007–2009
Outcome of the viva Engaged with Not engaged Total Average
research with research 2007–09
training training
n % n % n % n %
Awarded Ph.D. immediately 3 14 2 10 5 12 17 11
Awarded Ph.D. subject to minor 7 33 9 43 16 38 42 27
corrections submitted within 1 month
Awarded Ph.D. subject to minor 9 43 8 38 17 41 73 48
corrections submitted within 6 months
Revise and resubmit thesis within 12 2 10 1 5 3 7 14 9
months without a further oral
examination.
Revise and resubmit thesis within 12 0 0 1 5 1 2 7 5
months with a further oral examination.
The Impact on Submission of Doctoral Degrees

Total 21 100 21 100 42 100 153 100

Note: One candidate who had submitted was yet to sit the viva.

© 2011 The Authors. Higher Education Quarterly © 2011 Blackwell Publishing Ltd.
57
58 Higher Education Quarterly

Figure 1 T-shaped research postgraduate skills and expertise

Source: Leonard-Barton, 1995.

In order to help explain this result, Leonard-Barton’s (1995, p. 75)

work on the T-shaped manager was used. Leonard-Barton distinguishes
between generic skills such as problem solving and deep knowledge of an
application or area based on specialist skills. Figure 1 translates Leonard-
Barton’s concept of the T-shaped manager to the context of doctoral
research training; here the cross of the ‘T’ relates to the development of
transferable skills such as the ability to communicate effectively, work in
teams and manage research and careers and to the fostering of an
interdisciplinary awareness, all of which are the objectives of a centrally
organised research training programme. The deep disciplinary skills and
competencies represented by the upright of the ‘T’ are obtained through
discipline-based activities, such as seminar and conference attendance
and, most importantly, supervision, which is intense, regular and highly
focused on the subject matter of the individual thesis. The outcome of
the thesis, therefore, is more likely to be dependent on the development
of these deep disciplinary skills frequently obtained through supervision
than on the more generic, interdisciplinary and transferable skills devel-
oped through formal research training.

Multivariate analysis
The bivariate analyses above point to significant associations between
engagement with research training, submission of a PAF, supervisory

© 2011 The Authors. Higher Education Quarterly © 2011 Blackwell Publishing Ltd.
The Impact on Submission of Doctoral Degrees 59

arrangements and doctoral submission in four years. The analysis of the

cohort data is now taken further via a multivariate binary logistic regres-
sion appropriate for categorical data where the dependent variable is
dichotomous. Binary logistic regression allows a model to be built incor-
porating the key factors found to be associated with timely submission of
the thesis. This model is used to predict the probability of an event,
submission of thesis in this case, occurring given the predictor variables
mentioned above.
Before considering the predictions of probabilities, a brief description
of the results of the binary logistic regression analysis will be undertaken.
The inclusion of the three variables in the regression analysis increased
the predictive percentage of the model from 50.6% to 65.5%.The results
of the logistic regression analysis indicate that in the model the submis-
sion of a training programme assessment (TPA) variable is on the
borderline of significance at the 0.5 level, the submission of the PAF
variable is significant at the 0.10 level and the supervisor arrangements
(SA) variable is not significant (Table 4). Given the previous analysis,
however, all three variables were used to predict the probabilities of
submission of the Ph.D. thesis.
A new variable was computed consisting of all the various combina-
tions of submitting a PAF or not, submitting a TPA or not, and having
one or more than one supervisor (the three variables found to be asso-
ciated with submitting the thesis in earlier analysis).The way the analysis
was constructed meant that the different probabilities of thesis submis-
sion were calculated (Table 5).
This analysis produced striking results. For candidates who had nega-
tive scores for all three significant variables (not submitting a PAF, not
submitting a training programme assessment and having one supervisor)
the probability of thesis submission in four years is only 15%. For
candidates with positive results for all three variables (submitting a PAF

TABLE 4
Variables in the binary logistic regression equation
B S.E. Wald df Sig. Exp(B)
Step 1 a
PAF -1.198 .684 3.065 1 .080 .302
TPA -.899 .468 3.687 1 .055 .407
SA -.499 .589 .718 1 .397 .607
Constant 1.746 .674 6.711 1 .010 5.731
a. Variable(s) entered on step 1: submission of project approval form (PAF), submission of
a training programme assessment (TPA), supervisory arrangements (SA).

© 2011 The Authors. Higher Education Quarterly © 2011 Blackwell Publishing Ltd.
60 Higher Education Quarterly

TABLE 5
Predicted probability of submitting thesis by combinations of
significant factors
Probability No. of
cases
Not submitted PAF, not submitted TPA,1 supervisor .1485762 8
Not submitted PAF, not submitted TPA, supervisory team .2231868 3
Not submitted PAF, submitted TPA, 1 supervisor .3001114 3
Submitted PAF, not submitted TPA, 1 supervisor .3662635 7
Not submitted PAF, submitted TPA, supervisory team .4138320 3
Submitted PAF, not submitted TPA, supervisory team .4875878 34
Submitted PAF, submitted TPA, 1 supervisor .5868030 4
Submitted PAF, submitted TPA, supervisory team .7004383 25
Total .4942529 87
Key: PAF = project approval form; TPA = training programme assessment.

and assignments and having more than one supervisor) the probability
of thesis submission is increased to 70%. This latter category was the
second most common in the cohort, with 25 (29%) research candidates
falling into this group. The most common group consisted of research
candidates who had submitted a PAF and had a supervisory team but
had not engaged with the training programme (34; 39%); for these
candidates the probability of thesis submission was 49%. These results
provide a clear quantitative indication that research training and codes of
practice impact substantially on the probability of research students
submitting their Ph.D. in four years. The results point to the importance
of ensuring that doctoral students engage with research training and
university procedures across their programmes and that university
research degree regulations are geared towards ensuring this happens.

Conclusion
The main conclusion from this cohort study is that changes in the UK
doctorate, including the introduction of codes of research practice and
formal research training, have had an important impact on the prospects
of research candidates submitting their theses within the required time.
Three factors representing key elements of the changes have been found
to be significant in this respect and when combined to greatly improve
the chances of submission in four years, namely: submission of a PAF
outlining the research objectives and plan; completion of a training
programme assessment; and being supervised by a supervisory team.

© 2011 The Authors. Higher Education Quarterly © 2011 Blackwell Publishing Ltd.
The Impact on Submission of Doctoral Degrees 61

These factors increase the probability of the candidate submitting the

thesis within four years of registration from 15% to 70%.
In an era of greater austerity in higher education when institutional
budgets will be under threat, these findings provide important quantita-
tive evidence about the impact of skills training and development on
doctoral education, which is a major source of debate within contempo-
rary doctoral education in the UK, particularly since the study focuses
on outcomes of the Rugby Team Impact Framework (Bromley, 2010).
Also, surveying doctoral submission rates has long been the way in which
UK research councils monitor their research studentships; and univer-
sities in receipt of these studentships are keen to understand what
measures can be taken to avoid the sanctions resulting from falling below
the rates set (as noted above, currently 60% within four years set by the
ESRC and 70% by the Arts and Humanities Research Council). An
important challenge for the future will be to maintain recent improve-
ments in doctoral education while adapting current institution-specific
arrangements to meet the challenge of cross-institutional collaboration,
which will be increasingly encouraged by the research councils. These
results will also be of relevance to an international audience for whom
the UK experience is of considerable interest and influence, not least in
Europe where the third cycle of the Bologna Process is producing pro-
found changes in doctoral education across the 47 European countries
committed to the goals of the European Higher Education Area.
The analysis is based on the first cohort of doctoral candidates who
have experienced the changes to the British doctorate discussed and for
whom the outcome of their studies can be measured. Data will be
incorporated into the data set as more of the candidates in the current
cohort submit and complete their doctorates and subsequent cohorts
will be incorporated into the data set as time passes. This longitudinal
cohort methodology can be applied to other higher education
institutions. The initial findings of the study reported in this paper can
therefore be built on by more powerful statistical analysis of a more
complete data set, which will allow comparisons to be made both
between cohorts within and between institutions. Moreover, as data
from the Postgraduate Research Experience Survey amass over time, a
broader analysis should be possible of the influence on doctoral out-
comes of a wider range of factors associated with the student experience.
Acknowledgements
The authors would like to acknowledge the contributions of Dr Simon
Kometa for his statistical advice and Sarah Rylance, Caroline McLean

© 2011 The Authors. Higher Education Quarterly © 2011 Blackwell Publishing Ltd.
62 Higher Education Quarterly

and Gill McRae, members of the postgraduate team in the Faculty of

Humanities and Social Sciences, for their help in the production of this
paper.They would also like to thank the Editors and anonymous review-
ers for their very helpful comments.

References
Advisory Board for the Research Councils (ABRC) (1982) Report of the Working Party on
Postgraduate Education (Swinnerton-Dyer Report) Cmnd. 8537 (London: HMSO).
Austin, A., Kiley, M. and Pearson, M. (2009) Critical Issues in Doctoral Education:
Comparison of Challenges and Responses in Australia and the United States. Paper
presented at the American Society for Higher Education.
Booth, A. and Satchell, S. (1995) The Hazards of Doing a PhD—an Analysis of Comple-
tion and Withdrawal Rates of British PhDs in the 1980s, Journal of the Royal Statistical
Society Series A—Statistics in Society, 158, pp. 145–159.
Bowen, W. and Rudenstine, N. (1992) In Pursuit of a PhD. Princeton, NJ: Princeton
University Press.
Bromley, T. (2010) The Impact of Researcher Training and Development: Two Years On.
Cambridge: Careers Research and Advisory Group.
Bromley, T., Metcalfe, J. and Park C. (2008) The Rugby Team Impact Framework
http://www.vitae.ac.uk/CMS/files/1.Rugby Impact Framework_33.pdf, last accessed 22
December 2010
Council of Graduate Schools (2008) Ph.D. Completion and Attrition: Analysis of Baseline
Program Data from the Ph.D. Completion Project. Washington, DC: Council of Graduate
Schools.
Council of Graduate Schools (2009) Ph.D. Completion and Attrition: Findings from Exit
Surveys of Ph.D. Completers. Washington, DC: Council of Graduate Schools.
Council of Graduate Schools (2010) Ph.D. Completion and Attrition: Policies and Practices to
Promote Student Success. Washington, DC: Council of Graduate Schools.
Cryer, P. (2000) The Research Student’s Guide to Success. Buckingham: Open University
Press.
Cullen, D., Pearson, M., Saha, L. and Spear, R. (1994) Establishing Effective PhD
Supervision. Canberra: Higher Education Division, DEET.
Davis, H., Evans, T. and Hickey, C. (2006) A Knowledge-based Economy Landscape:
Implications for Tertiary Education and Research Training in Australia, Journal of
Higher Education Policy and Management, 28 (3), pp. 213–244.
Economic and Social Research Council (ESRC) (2010) The Importance of PhD
Submission Rates, available at http://www.esrcsocietytoday.ac.uk/ESRCInfoCentre/
opportunities/postgraduate/training/recognition/submissions.aspx, last accessed 22
December 2010.
European Universities Association (EUA) (2005) Doctoral Programmes for the European
Knowledge Society. Brussels: European Universities Association.
European Universities Association (EUA) (2007) Doctoral Programmes in Europe’s Univer-
sities: Achievements and Challenges. Report prepared for European universities and
ministers of higher education. Brussels: European Universities Association.
Goldberger, M., Maher, B. and Flattau, P. (eds.) (1995) Research-doctorate Programs in
the United States: Continuity and Change. Washington, DC: National Academies
Press.
Golde, C. and Dore, R. (2004) The Survey of Doctoral Education and Career Preparation:
the Importance of Disciplinary Contexts. In D. Wulff and A. Austin (eds.) Path to the
Professoriate: Strategies for Enriching the Preparation of Future Faculty. San Francisco, CA:
Jossey-Bass.

© 2011 The Authors. Higher Education Quarterly © 2011 Blackwell Publishing Ltd.
The Impact on Submission of Doctoral Degrees 63

Harris, M (1996) Review of Postgraduate Education: Report. Bristol: Higher Education

Funding Council for England, Committee of Vice-Chancellors and Principals and
Standing Conference of Principals.
Higher Education Funding Council for England (2007) Research Degree Qualification
Rates http://www.hefce.ac.uk/pubs/hefce/2007/07_29/, last accessed 22 December 2010
Higher Education Funding Council for England (2010) Research Degree Qualification
Rates http://www.hefce.ac.uk/pubs/hefce/2010/10_21/, last accessed 22 December 2010
Joint Declaration of the European Ministers of Education (1999) The Bologna Declara-
tion: http://www.bologna-bergen2005.no/Docs/00-Main_doc/990719BOLOGNA_
DECLARATION.PDF last accessed 22 December 2010.
Kehm, B. (2005) Developing Doctoral Degrees and Qualifications in Europe. Good
Practice and Issues of Concern, Contributions to the Higher Education Research, 27 (1),
pp. 10–33.
Kehm, B. (2007) Quo Vadis Doctoral Education? New European Approaches in the
Context of Global Changes’, European Journal of Education, 42 (3), pp. 307–319.
Kehm, B. (2010) The Future of the Bologna Process: the Bologna Process of the Future.
Editorial, European Journal of Education, 45 (4), pp. 529–534.
Kemp, D. (1999a) Knowledge and Innovation: A Policy Statement on Research and Research
Training (White Paper). Canberra: Government of Australia.
Kemp, D. (1999b) New Knowledge, New Opportunities: A Discussion Paper on Higher Edu-
cation Research and Research Training. Canberra: DETYA.
Kiley, M. and Mullins, G. (eds.) (2006) Quality in Postgraduate Research: Knowledge
Creation in Testing Times. Canberra: CEDAM, The Australian National University.
Leonard-Barton D. (1995) Wellsprings of Knowledge: Building and Sustaining the Sources of
Innovation. Boston, MA: Harvard Business School Press.
McWilliam, E. and James, R. (2002) Doctoral Education in a Knowledge Economy.
Editorial, Higher Education Research & Development, 21 (2), p. 117.
Nerad, M. (2006) Globalisation and Its Impact on Research Education: Trends and
Emerging Best Practices for the Doctorate of the Future. In M. Kiley and G. Mullins
(eds.), Quality in Postgraduate Research: Knowledge Creation in Testing Times. Canberra:
CEDAM, The Australian National University, pp. 11–12
Nerad, M. and Heggelund, M. (2008) Toward a Global PhD? Forces and Forms in Doctoral
Education Worldwide. Seattle, WA: University of Washington Press.
Nyquist, J. and Woodford, B. (2000) Re-envisioning the Ph.D.:What Concerns Do We Have?
Seattle: University of Washington, Center for Instructional Development and Research.
Park, C. (2009) The Research Student Experience: Lessons from PRES. York: Higher Educa-
tion Authority.
Pearson, M. (2005) Framing Research on Doctoral Education in Australia in a Global
Context, Higher Education and Research and Development, 24 (2), pp. 119–134.
Pearson, M., Evans, T. and Macauley, P. (2008) Growth and Diversity in Doctoral
Education: Assessing the Australian Experience, Higher Education, 55, pp. 357–
372.
Quality Assurance Agency for Higher Education (QAA) (2004) Code of Practice for
the Assurance of Academic Quality and Standards in Higher Education, Section
1 Postgraduate Research Programmes http://www.qaa.ac.uk/academicinfrastructure/
codeOfPractice/section1/postgrad2004.pdf, last accessed 22 December 2010
Research Councils United Kingdom (RCUK) (2002) Joint Statement of the Research
Councils’ Skills Training Requirements for Research Students http://www.rcuk.ac.uk/
documents/researchcareers/jsstrainingrequirements.pdf, last accessed 22 December
2010
Research Councils United Kingdom (RCUK) (2010) Review of progress in implementing
the recommendations of Sir Gareth Roberts, regarding employability and career
development of PhD students and research staff http://www.rcuk.ac.uk/documents/
researchcareers/RobertReport2011.pdf, last accessed 25 January 2011

© 2011 The Authors. Higher Education Quarterly © 2011 Blackwell Publishing Ltd.
64 Higher Education Quarterly

Roberts, G. (2001) SET for success: The supply of people with science, technology,
engineering and mathematics skills http://webarchive.nationalarchives.gov.uk/+/http://
www.hm-treasury.gov.uk/d/robertsreview_introch1.pdf last accessed 22 December
2010
Robbins, L. (1963) Higher Education: Report of the Commission on Higher Education under the
Chairmanship of Lord Robbins, 1961–63. London: HMSO.
Sinclair, M. (2004) The Pedagogy of ‘Good’ PhD Supervision: A National Cross-disciplinary
Investigation of PhD Supervision. Canberra: Department of Education and Training.
Tinkler, P. and Jackson, C. (2000) Examining the Doctorate: Institutional Policy and the
PhD Examination Process in the UK, Studies in Higher Education, 25, 167–180.
Tinkler, P. and Jackson, C. (2002) In the Dark? Preparing for the PhD Viva, Quality
Assurance in Education: Special Issue, Quality and Standards in Doctoral Awards, 10 (2),
pp. 86–97.
Tinkler, P. and Jackson, C. (2004) The Doctoral Examination Process: A Handbook for
Students, Examiners and Supervisors. Maidenhead: Open University Press.
Vitae (2010) Researcher Development Statement. http://www.vitae.ac.uk/policy-practice/
275981/Researcher-Development-Statement-RDS.html, last accessed 22 December
2010
Welsh, I. (1980) The Postgraduate Student: Progress and Problems, PhD thesis, University of
Aberdeen.
West, R. (1998) Learning for Life Final Report: Review of Higher Education Financing
and Policy (http://www.deetya.gov.au/divisions/hed/hereview/default.htm). Canberra:
DEETYA.
Winfield, G. (1987) The Social Science PhD, the ESRC Enquiry into Submission Rates: The
Report. London: Economic and Social Research Council.
Wright, T. and Cochrane, R. (2000) Factors Influencing Successful Submission of PhD
Theses, Studies in Higher Education, 25 (2), pp. 182–195.

© 2011 The Authors. Higher Education Quarterly © 2011 Blackwell Publishing Ltd.