Beruflich Dokumente
Kultur Dokumente
com
Open Access
Research
ABSTRACT
Objectives: To study whether systematic reviewers
apply procedures to counter-balance some common
forms of research malpractice such as not publishing
completed research, duplicate publications, or selective
reporting of outcomes, and to see whether they
identify and report misconduct.
Design: Cross-sectional analysis of systematic reviews
and survey of their authors.
Participants: 118 systematic reviews published in
four journals (Ann Int Med, BMJ, JAMA, Lancet), and
the Cochrane Library, in 2013.
Main outcomes and measures: Number (%) of
reviews that applied procedures to reduce the impact
of: (1) publication bias (through searching of
unpublished trials), (2) selective outcome reporting (by
contacting the authors of the original studies), (3)
duplicate publications, (4) sponsors and (5) authors
conflicts of interest, on the conclusions of the review,
and (6) looked for ethical approval of the studies.
Number (%) of reviewers who suspected misconduct
are reported. The procedures applied were compared
across journals.
Results: 80 (68%) reviewers confirmed their data. 59
(50%) reviews applied three or more procedures; 11
(9%) applied none. Unpublished trials were searched
in 79 (66%) reviews. Authors of original studies were
contacted in 73 (62%). Duplicate publications were
searched in 81 (69%). 27 reviews (23%) reported
sponsors of the included studies; 6 (5%) analysed
their impact on the conclusions of the review. Five
reviews (4%) looked at conflicts of interest of study
authors; none of them analysed their impact. Three
reviews (2.5%) looked at ethical approval of the
studies. Seven reviews (6%) suspected misconduct;
only 2 (2%) reported it explicitly. Procedures applied
differed across the journals.
Conclusions: Only half of the systematic reviews applied
three or more of the six procedures examined. Sponsors,
conflicts of interest of authors and ethical approval remain
overlooked. Research misconduct is sometimes identified,
but rarely reported. Guidance on when, and how, to report
suspected misconduct is needed.
INTRODUCTION
Rationale
Research misconduct can have devastating
consequences for public health1 and patient
care.2 3 While a common denition of
research misconduct is still lacking, there is
an urgent need to come up with strategies to
prevent it.4 5 Fifteen years ago, Smith6 proposed a preliminary taxonomy of research
misconduct describing 15 practices ranging
from minor to major misconduct. Some of
these practices, however, are very common
and may not be regarded by all as misconduct. Therefore, this research denes malpractice as relatively common and minor
misconduct, while the term misconduct is
used for data fabrication, falsication, plagiarism
or
any
other
intentional
malpractices.
It has been shown that some of the 15 malpractices described by Smith, threaten the
conclusions of systematic reviews. Examples
of such malpractices include: avoiding the
publication of a completed research,7 8
Open Access
duplicate publications,9 selectively reporting on outcomes or adverse effects,10 and presenting biased results
that are in favour of the sponsors11 or the authors12
interests. The impact of other malpractices, such as gift
or ghost authorship, is less clear.
Rigorous systematic review methodology includes specic procedures that can counter-balance some of the
research malpractice. Unpublished studies may, for
example, be identied through exhaustive literature
searches,13 and statistical tests or graphical displays such
as funnel plots can quantify the risk of publication
bias.14 Unreported outcomes may be unearthed by contacting the authors of original articles, and multiple publications based on the same cohort of patients can be
identied and excluded from analyses.15 Authors of systematic reviews (further called: systematic reviewers or
reviewers) can also use sensitivity analyses to quantify the
impact sponsors and authors personal interests have on
the conclusions of a review. Finally, it has been suggested
that, as part of the process of systematic reviewing,
ethical approval of included studies or trials should be
checked on in order to identify unethical research.16 17
Systematic reviewers could hence act as whistle-blowers
when reporting any suspected misconduct.18
Objectives
The aim of this study is to examine whether systematic
reviewers apply the aforementioned procedures, and
whether they uncover and report on cases of
misconduct.
The study rst examines whether reviewers searched
for unpublished studies or tested for publication bias,
contacted authors to unearth unreported outcomes,
searched for duplicate publications, analysed the impact
of sponsors or possibles conicts of interest of study
authors, checked on ethical approval of the studies and
reported on misconduct. The secondary objective was to
examine whether four major journals and the Cochrane
Library reported consistently on the issue.
METHODS
The reporting of this cross-sectional study follows the
STROBE recommendation.19 The protocol is available
from the authors.
Study design
We conducted a cross-sectional analysis of systematic
reviews published in 2013 in four general medical journals (Annals of Internal Medicine (Ann Int Med), The BMJ
(BMJ), JAMA and The Lancet (Lancet)). A random
sample of new reviews was drawn from the Cochrane
Database of Systematic Reviews (Cochrane Library) in 2013,
as Cochrane reviews are considered the gold standard in
terms of systematic reviewing.
2
Open Access
Table 1 Rating of the six procedures examined
Score
0
Search of unpublished
trials and/or test for
publication bias
Contact with study authors
Duplicate publications
Ethical approval of
included studies not
reported
RESULTS
Selection of reviews
We identied 136 references; 18 were excluded for different reasons, leaving us with 118 systematic reviews
(Ann Int Med 39A1-39; BMJ 38B1-38; JAMA 12J1-12; Lancet
Elia N, et al. BMJ Open 2016;6:e010442. doi:10.1136/bmjopen-2015-010442
Open Access
Sponsors
Twenty-seven reviews (23%) reported on the sources of
funding for the studies. Six reviewers (5%) analysed the
Elia N, et al. BMJ Open 2016;6:e010442. doi:10.1136/bmjopen-2015-010442
Open Access
Table 2 Characteristics of the systematic reviews analysed
ALL
N
118
Per cent
Answer
N
Per cent
No answer
N
Per cent
100
80
38
68
p Value
32
0.240
6 (59)
6 (47.5)
6 (59)
0.969
42
23
21
4
28
36
19
18
3
24
27
16
14
3
20
34
20
18
4
25
15
7
7
1
8
39
18
18
3
21
82
3
33
69
3
28
53
2
25
66
3
31
29
1
8
76
3
21
8
24
86
7
20
73
7
15
58
9
19
73
1
9
28
3
24
74
0
42
61
15
0
36
52
13
0
33
39
8
0
41
49
10
0
9
22
7
0
24
58
18
0.513
0.420
0.130
0.637
4 (37)
4 (37)
4 (36)
0.569
56
42
7
13
47
36
6
11
35
30
6
9
44
38
8
11
21
12
1
4
55
32
3
11
0.804
28 (1257)
28 (1259)
34 (1255)
46
5
5
62
32
4
5
39
14
1
0
23
0.312
39
4
4
53
40
5
6
49
37
3
0
61
Answer: reviews in which extracted data were confirmed by reviewers. No answer: reviews in which extracted data were not confirmed by
reviewers. p Value testing the null hypothesis of equal distribution between the reviews for which the author responded to our inquiries and
those who did not. Statistical tests: 2 test or Kruskal-Wallis equality of population rank tests, as appropriate.
RCT, randomised controlled trials.
(8,14)
Open Access
Table 3 Application of the procedures to counter-balance some common research malpractices
ALL
N
Per cent
Answer
N
Per cent
No answer
N
Per cent
118
100
80
68
38
32
39
56
23
33
47
19
26
39
15
33
49
19
13
17
8
34
45
21
45
15
58
38
13
49
28
12
40
35
15
50
17
3
18
45
8
47
37
71
10
31
60
8
21
54
5
26
68
6
16
17
5
42
45
13
91
21
6
77
18
5
63
13
4
79
16
5
28
8
2
74
21
5
113
5
0
96
4
0
77
3
0
96
4
0
36
2
0
95
5
0
115
3
0
97
3
0
77
3
0
96
4
0
38
0
0
100
0
0
6
8
34 43
38 48
2
3
2.5 (23)
5
14
18
1
2.5
13
37
47
3
(13)
74
6
37
1
p Value
0.914
0.427
0.057
0.809
0.703
0.481
0.403
11
9
48
41
56
47
3
3
2.5 (13)
0.296
111
7
94
6
93
8
97
3
Answer: reviews in which extracted data were confirmed by reviewers. No answer: reviews in which extracted data were not confirmed by
reviewers. p Value testing the null hypothesis of equal distribution between the reviews for which the authors responded to our inquiry and
those who did not (2 test). Percentages may not add-up to 100% because of rounding errors.
Suspicion of misconduct
Two reviewers suspected research misconduct in the articles included in their review and reported it
accordingly.B31,J12 Contacting the other reviewers
allowed us to uncover ve additional cases of possible
misconduct. Four agreed to be cited here;A26,B33,C16,L1
while one preferred to remain anonymous.
Data falsication was suspected in three reviews.A26,C16,
J12
One review, looking at the association of hydroxyethyl
starch administration with mortality or acute kidney
injury of critically ill patients,J12 included seven articles
co-authored by Joachim Boldt. However, a survey performed in 2010, and focusing on Boldts research published between 1999 and 2010, had led to the retraction
of 80 of his articles due to data fabrication and lack of
ethical approval.20 The seven articles co-authored by
Boldt were kept in the review as they had been published before 1999. Nonetheless, the reviewers performed
sensitivity analyses excluding these seven articles, and
6
Open Access
Intentional selective reporting of outcomes was suspected in two reviews.B31 In one review that examined
early interventions to prevent psychosis, the reviewers
identied, discussed and referenced three articles that
did not report on all the outcomes.B31 In the second
review, the corresponding reviewer (who preferred to
remain anonymous) revealed that he knew of two situations in which authors knew what the results showed if
they used standard categories (of outcome) but did not
publish them because there was no relationship, or not
the one they had hoped to nd.
Plagiarism was identied in one review examining
the epidemiology of Alzheimers disease and other
forms of dementia in China.L1 According to the
reviewers, they had identied a copy-paste-like duplicate publication of the same paper published two or
more times, with the same results and sometimes even
different authors. This information was not reported
in the published review. The corresponding reviewer
explained that they had a brief discussionabout
what to do about those ndings and whether to
mention them in the paperWe did not think that we
should be distracted from our main goal, so we felt
that it was better to leave it to qualied bodies and
specialised committees on research malpractice to
address this problem separately.
Finally, one reviewer told us that there were some
suspected misconduct in original studies on which we
performed sensitivity analysis (best caseworst case).
The review mentioned that some studies were of poor
quality, but it did not specically mention suspicion of
misconduct.B33
The median number of studies included in the
reviews that detected misconduct was 56 (IQR, 1197),
and was 28 (IQR, 1257) in the reviews that did not
detect misconduct. The difference did not reach statistical signicance.
Secondary endpoints
The reviews published in the four medical journals differed in most characteristics examined in this paper. The
reviews published in the Cochrane Library differed from all
the other reviews (see online supplementary appendix
table 2A). There were also differences in procedures
applied across the journals (see online supplementary
appendix table 4A). The only three reviews that had
extracted data on ethical approval for the studies were
published in the BMJ (2) and in the Cochrane Library (1).
Finally, one review from the BMJ and one review from
JAMA explicitly mentioned potential misconduct.
DISCUSSION
Statement of principal findings
This analysis conrms some issues and highlights new
ones. The risk related to double counting of participants
due to duplicate publications and the risk of selective
reporting of outcomes are reasonably well recognised.
Elia N, et al. BMJ Open 2016;6:e010442. doi:10.1136/bmjopen-2015-010442
Open Access
process of systematic reviewing.16 However, our study
shows that only three reviews reported having done so.
The need to report ethical approval in original studies
has only recently been highlighted. A case of massive
fraud in anaesthesiology,20 in which informed consent
from patients and formal approval by ethics committee
were fabricated, only shows how difcult a task it will be.
The most striking nding was that although seven systematic reviews suspected misconduct in original studies,
ve of them did not report it, one reported it without
further comment and only one reported overtly on the
suspicion. This illustrates that reviewers do not consider
themselves entitled to make an allegation, although they
are in a privileged position to identify misconduct. The
fact that one reviewer preferred to remain anonymous
further illustrates the reluctance to openly report on
misconduct.
Strengths and weaknesses
We used a clear and reproducible sampling method that
was not limited to any medical specialty. The data analysed had been conrmed by the reviewers. This allowed
us to quantify the proportion of procedures that were
implemented but not reported by the reviewers. Finally,
to our knowledge, this is the rst analysis of the procedures used by systematic reviewers to deal with research
malpractice and misconduct. Qualitative answers of the
reviewers were very informative.
We selected systematic reviews from four major
medical journals and the Cochrane Library, which is considered the gold standard in terms of systematic reviewing. We can reasonably assume that the problems
identied are at least as serious in other medical journals. Systematic reviews that were not identied as such
in their titles were not included. However, including
these reviews would not have changed our ndings.
Only one reminder was sent to the reviewers, since the
response rate was reasonably high. Furthermore, the
characteristics of the systematic reviews did not differ
between reviews for which authors responded or not.
We did not ask the reviewers to conrm all their data
but focused on the information that we considered
unclear. It is possible that some of the reviewers had
indeed extracted information on sponsors and authors
conicts of interest, as well as ethical approval of the
studies, but failed to report them. A number of procedures applied and instances of misconduct came to light
through our personal contacts with the reviewers. Our
results might therefore be underestimated. On the other
hand, it is possible that some reviewers pretended having
applied some procedures although they had not. This
would have led to an overestimation of the number of
procedures applied. The major weakness of this study
lies in the lack of an accepted denition of research
misconduct. It is possible that some reviewers might
have hesitated to disclose suspected misconduct, leading
to an underestimation of the prevalence of misconduct
identied. Finally, we have used the preliminary
8
Open Access
2
12.
13.
14.
15.
16.
17.
18.
19.
20.
21.
22.
23.
24.
25.
26.
27.
28.
REFERENCES
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
Deer B. How the case against the MMR vaccine was fixed. BMJ
2011;342:c5347.
Bouri S, Shun-Shin MJ, Cole GD, et al. Meta-analysis of secure
randomised controlled trials of -blockade to prevent perioperative
death in non-cardiac surgery. Heart 2014;100:45664.
Krumholz HM, Ross JS, Presler AH, et al. What have we learnt from
Vioxx? BMJ 2007;334:120.
Martinson BC, Anderson MS, de Vries R. Scientists behaving badly.
Nature 2005;435:7378.
Lscher TF. The codex of science: honesty, precision, and truth
and its violations. Eur Heart J 2013;34:101823.
Smith R. The COPE Report 2000. What is research misconduct?
http://publicationethics.org/files/u7141/COPE2000pdfcomplete.pdf
(accessed 26 Oct 2015).
Eyding D, Lelgemann M, Grouven U, et al. Reboxetine for
acute treatment of major depression: systematic review and
meta-analysis of published and unpublished placebo and
selective serotonin reuptake inhibitor controlled trials. BMJ
2010;341:c4737.
Hart B, Lundh A, Bero L. Effect of reporting bias on meta-analyses
of drug trials: reanalysis of meta-analyses. BMJ 2012;344:d7202.
Tramr MR, Reynolds DJ, Moore RA, et al. Impact of covert
duplicate publication on meta-analysis: a case study. BMJ
1997;315:63540.
Selph SS, Ginsburg AD, Chou R. Impact of contacting study authors
to obtain additional data for systematic reviews: diagnostic accuracy
studies for hepatic fibrosis. Syst Rev 2014;3:107.
Huss A, Egger M, Hug K, et al. Source of funding and results of
studies of health effects of mobile phone use: systematic
review of experimental studies. Environ. Health Perspect
2007;115:14.
29.
30.
31.
32.
33.
34.
35.
36.
37.
38.
Open Access
39.
40.
10
41.
42.
doi: 10.1136/bmjopen-2015-010442
Updated information and services can be found at:
http://bmjopen.bmj.com/content/6/3/e010442
These include:
References
This article cites 37 articles, 15 of which you can access for free at:
http://bmjopen.bmj.com/content/6/3/e010442#BIBL
Open Access
Email alerting
service
Receive free email alerts when new articles cite this article. Sign up in the
box at the top right corner of the online article.
Topic
Collections
Notes