Urologic Oncology: Seminars and Original Investigations 33 (2015) 387.e1–387.

e6

Original article
Multi-institutional external validation of urinary TWIST1 and NID2
methylation as a diagnostic test for bladder cancer
Joseph J. Fantony, M.D.a, Michael R. Abern, M.D.b, Ajay Gopalakrishna, B.S.a,
Richmond Owusu, M.D.a, Kae Jack Tay, M.B.B.S.a, Raymond S. Lance, M.D.c,
Brant A. Inman, M.D., M.S.a,*
a
Division of Urology, Duke University Medical Center, Durham, NC
b
Department of Urology, University of Illinois Chicago, Chicago, IL
c
Division of Urology, Urology of Virginia, Virginia Beach, VA
Received 12 February 2015; received in revised form 4 April 2015; accepted 24 April 2015

Abstract
Objectives: We previously reported a clinical trial in which we were unable to replicate the excellent diagnostic metrics produced in the
developmental study of the TWIST1 and NID2 gene methylation assay. In this expanded trial with subjects enrolled from another institution,
we reexamine the diagnostic capabilities of the test to externally validate our previous study.
Materials and methods: TWIST1 and NID2 gene methylation was assessed in DNA isolated from the urine of subjects at risk of bladder
cancer undergoing cystoscopy for hematuria or bladder cancer surveillance. The diagnostic gold standard was cystoscopy. Two thresholds of
TWIST1 and NID2 gene methylation were used for determining test result positivity, those published by Renard et al. and Abern et al. The
sensitivity, specificity, positive and negative predictive values, diagnostic likelihood ratios, and receiver operating characteristic curves were
calculated for each gene, as well as their combination. In all, 3 methods were used to combine TWIST1 and NID2 into a single composite test:
(1) believe-the-positive decision rule—if either gene is methylated the test result is positive, which maximizes test sensitivity; (2) believe-the-
negative decision rule—if either gene is not methylated the test result is negative, which maximizes test specificity; and (3) a likelihood-based
logistic regression model approach that balances sensitivity and specificity. Clinical utility was determined using a decision curve analysis.
Results: A total of 209 subjects were evaluated: 40% for hematuria and 60% for bladder cancer surveillance. Approximately 75% were
male, most of the prior cancers being low-grade Ta. Using cystoscopy as the gold standard, areas under the curve were 0.67 for TWIST1,
0.64 for NID2, and 0.66 for combined TWIST1 and NID2. Decision rule results revealed optimization of sensitivity at 67% using Renard
thresholds and specificity using the Abern thresholds at 69%. We found improved sensitivity (78%) in current smokers. Decision curve
analyses revealed that the methylation assay provided only a modest benefit even at high probabilities of missed cancer.
Conclusion: A urine DNA test measuring TWIST1 and NID2 methylation was externally examined with a larger cohort and its results
continue to be poor. These 2 biomarkers are unlikely to replace cystoscopy, but they may be worthy of study in active smokers. r 2015
Elsevier Inc. All rights reserved.

Keywords: Diagnostic test; Epigenetic; Sensitivity; Specificity; Urothelial carcinoma; Smoking

1. Introduction malignancy owing to its high recurrence rate and frequent

Bladder cancer is the fourth most common cancer in
the United States and the most expensive per patient
This study was funded, in part, by MDxHealth (formerly OncoMeth-
lyome Sciences).
* Corresponding author. Tel.: þ1-919-681-8760; fax: þ1-919-684-5220.
E-mail address: brant.inman@duke.edu (B.A. Inman).
need for surveillance [1]. In 2014, in the United States, the
estimated incidence of bladder cancer was 74,690 and
accounted for 15,580 deaths. Combined cystoscopy and
cytology have traditionally been considered to be the gold
standard for diagnosis and follow-up of bladder tumors,
despite significant limitations [2]. Cystoscopy is highly
sensitive but is invasive, uncomfortable, and occasionally
distressing to patients [3]. Moreover, the performance of

http://dx.doi.org/10.1016/j.urolonc.2015.04.014
1078-1439/r 2015 Elsevier Inc. All rights reserved.
387.e2 J.J. Fantony et al. / Urologic Oncology: Seminars and Original Investigations 33 (2015) 387.e1–387.e6
cystoscopy is operator-dependent and small tumors can be
missed. Cytology has traditionally been reported to have a
high specificity (86%96%), but it lacks sensitivity, espe-
cially for low-grade lesions [4,5]. Thus, there is a need for a
noninvasive urinary marker with high specificity and sensi-
tivity to detect both high- and low-grade bladder tumors.
Aberrant methylation of genes has been implicated in
several human cancers [6,7]. Renard et al. [8] had reported
the results of a novel urinary methylation test, which
identified 2 genes, TWIST1 and NID2, which, when methy-
lated, offer high sensitivity (90%) and specificity (93%) in
the detection of urothelial carcinoma. We subsequently
performed a validation study and reported a significantly
lower sensitivity (79%) and specificity (63%) [9]. The
marked difference in test performance between the initial
Renard development study and the external validation study
warrants additional investigation. Therefore, the purpose of
this study is to reevaluate the diagnostic accuracy of the
TWIST1 and NID2 urine methylation assay by examining a
second validation cohort of subjects from another institution.
2. Patients and methods
2.1. Patients
After institutional review board approval, subjects under-
going cystoscopy for either hematuria evaluation or non-
muscle invasive bladder cancer (NMIBC) surveillance were
enrolled in this prospective clinical trial between 9/2008 and
12/2010 at 2 independent institutions, Duke University
Medical Center (DUMC) and the Eastern Virginia Medical
School (EVMS). On enrollment patients were assigned to 1
of 3 groups: (1) those undergoing initial hematuria workup,
(2) those undergoing surveillance of NMIBC, and (3) those
with NMIBC being treated with BCG. Patients with a history
of NMIBC were eligible if (1) their tumor had previously
been completely transurethrally resected (including re-
resection in patients with T1 tumors) and (2) at least 1 year
had elapsed since treatment with BCG (induction or main-
tenance). All patients underwent evaluation of the upper
tracts as was appropriate as per the current standard of care.
Consenting patients underwent standard-of-care flexible cys-
tourethroscopy to determine the presence or absence of a
bladder tumor. Voided urine was collected with 30 ml
allocated for DNA methylation testing and the remainder
allocated for urine cytology if clinically indicated.
2.2. Urine TWIST1 and NID2 methylation tests
Urine was collected in a sterile fashion, frozen to 201C
within 1 hour of collection, and shipped on dry ice to a
central laboratory where genomic DNA was isolated and
analyzed with real-time methylation-specific polymerase
chain reaction (MS-PCR). The central laboratory was
blinded to patient clinical information and outcomes.
Laboratory procedures used for these samples were identi-
cal to those used by Renard et al., and detailed information
about the development and validation of the MS-PCR
assays used is published elsewhere [8]. Briefly, MS-PCR
was performed using parallel amplification and quantifica-
tion processes using specific primers and probes for NID2,
TWIST1, and β-actin on an ABI PRISM 7900HT instru-
ment (Applied Biosystems, Foster City). Sequence Detec-
tion System v.2.2 software was then used to calculate
methylated gene copy numbers. To be considered adequate
for study each sample needed to have at least 10 copies of
the β-actin gene in the isolated urinary DNA. Gene copy
numbers of methylated NID2 and TWIST1 were measured
on a continuous scale, but were later dichotomized as a
methylated or not using the thresholds of Renard et al. and
Abern et al. [8,9].
2.3. Cystoscopy as the diagnostic gold standard for
bladder cancer
Each subject underwent white light cystoscopy within 14
days of urine collection (usually the same day). Cystoscopy
results were classified as positive (i.e., obvious bladder
cancer), suspicious (i.e., abnormal but unclear if bladder
cancer), or negative (i.e., no evidence of bladder cancer).
White light cystoscopy is currently the community gold
standard for determining the presence or absence of bladder
cancer. Blue light fluorescent and narrow band imaging
cystoscopies are more sensitive but less specific methods of
cystoscopy, but they were not commercially available
during the study period [10].
2.4. Statistical methods
TWIST1 and NID2 distributions were inspected with
histograms and kernel density plots, demonstrating that they
were both skewed due to the presence of zero gene copy
numbers. Both were transformed (log(x þ 1)) to normality
for regression models [11]. ROC curves were constructed
for each methylated gene test with the cystoscopy gold
standard and the areas under the curve (AUC) calculated
with the trapezoidal method and 95% CI with the method of
DeLong [12,13]. The performance of TWIST1 and NID2
was summarized with standard diagnostic test metrics (i.e.,
sensitivity, specificity, predictive values, and diagnostic
likelihood ratios) from data dichotomized along the pub-
lished test thresholds of Renard et al. (TWIST1 Z 8 or
NID2 Z 30) and Abern et al. (TWIST1 Z 37 or
NID2 Z 21) [8,9]. As an exploratory analysis, diagnostic
test metrics were computed for several clinically relevant
subgroups of the cohort based on cigarette smoking (current
smoker, ex-smoker, or never smoker), and bladder cancer
history (no prior bladder cancer, prior non-muscle-invasive
bladder cancer, or prior BCG intravesical therapy).
TWIST1 and NID2 were combined into a single test
using 3 alternative methods: (1) believe-the-positive rule
 J.J. Fantony et al. / Urologic Oncology: Seminars and Original Investigations 33 (2015) 387.e1–387.e6 387.e3

Table 1 EVMS. Overall, 40% of patients were undergoing a hematuria

Baseline demographic characteristics workup and 60% were undergoing surveillance cystoscopy for
Duke (n ¼ 111) EVMS (n ¼ 98) NMIBC. Most of the patients with a prior history of NMIBC
were of stage Ta (78%), and most of the prior tumors were of
Gender
low grade (58%). Positive cystoscopies were noted in 25% of
Male 80 72% 76 78%
Female 31 28% 22 22% the cohort during the study period. The remaining characteristics
Age of both cohorts can be found in Table 1.
Median, IQR 67.3 56.4, 76.2 69.5 62.3, 77.0
Tobacco smoking 3.1. Assay performance
Current 12 11% 11 11%
Former 61 55% 36 37%
Never 37 33% 18 18% The median urine β-actin gene copy number was 8,362
Unknown 1 1% 33 34% (interquartile range [IQR]: 1,56244,880) whereas the
Indication for cystoscopy median urinary TWIST1 and NID2 gene copy numbers
Hematuria 53 48% 30 31% were 4.71 (IQR: 082.96) and 3.22 (IQR: 0.1619.94),
Bladder cancer surveillance 58 52% 68 69%
Most recent T stage
respectively. Table 2 summarizes test performance charac-
Tis 4 7% 5 9% teristics using both the Renard et al. and Abern et al.
Ta 41 71% 57 83% positive test thresholds under the believe-the-positive
T1 13 22% 6 9% (Table 2A) and believe-the-negative (Table 2B) Boolean
Most recent tumor grade test interpretation rules. The Renard et al. thresholds were
Low grade 35 60% 38 56%
High grade 22 38% 27 40%
more sensitive but less specific than the Abern et al.
Unknown 1 2% 3 4% thresholds under the believe-the-positive rule, but these
Any prior BCG 25 43% 38 56% differences were not statistically significant. Under the
Study cystoscopy believe-the-negative rule, both thresholds had similar diag-
Positive 29 26% 23 23% nostic performance. These results imply that the methylated
Suspicious 8 7% 4 4%
Negative 74 67% 71 70%
gene copy threshold used to define a positive test result
(i.e., Renard vs. Abern) did not have an important influence
on ultimate test performance. However, and as expected,
(i.e., if either gene's copy number was above the specified test specificity was maximized using the believe-the-
threshold then the combined test was positive) [14], (2) negative rule whereas test sensitivity was maximized when
believe-the-negative rule (i.e., if either gene's copy number using the believe-the-positive rule. As the urine DNA
was below the specified threshold then the combined test methylation test evaluated in this manuscript is being
was negative) [14], and (3) a risk score approach based on investigated as a replacement for cystoscopy, its perform-
a logistic regression combination of TWIST1 and NID2 ance under the believe-the-positive rule (which maximizes
[15]. A combined ROC curve was constructed using a sensitivity) is most important and the sensitivity was
likelihood approach, smoothed using a binormal model, and unacceptably low at 67% (Renard thresholds) and 58%
the 95% CI of the AUC was determined using 2,000 (Abern thresholds).
bootstrap replicates [16].
The clinical utility of the combined methylation assay Table 2
Diagnostic performance of the urine DNA methylation assay
was determined using a decision curve analytic approach
[17]. Using this approach the net clinical benefit was plotted Renard thresholds Duke thresholds
against the threshold probability of missing a bladder cancer (TWIST1 Z 8 or (TWIST1 Z 37 or
for 3 models: (1) cystoscopy for all patients with hematuria, NID2 Z 30) NID2 Z 21)
(2) cystoscopy for no patients, and (3) the risk score logistic (A) Believe the positive
regression model combining TWIST1 and NID2. All Sensitivity 0.67 (0.53, 0.80) 0.58 (0.43, 0.71)
statistical analyses were performed using R version 3.0.3 Specificity 0.61 (0.52, 0.68) 0.69 (0.62, 0.77)
PPV 0.36 (0.27, 0.46) 0.38 (0.28, 0.50)
(R Foundation for Statistical Computing, Vienna, Austria)
NPV 0.85 (0.77, 0.91) 0.83 (0.76, 0.89)
with packages ROCR, epiR, and pROC installed [18,19]. Positive DLR 1.70 (1.30, 2.23) 1.89 (1.35, 2.63)
Negative DLR 0.54 (0.36, 0.81) 0.61 (0.44, 0.85)
(B) Believe the negative
3. Results Sensitivity 0.37 (0.24, 0.51) 0.35 (0.22, 0.49)
Specificity 0.86 (0.80, 0.91) 0.85 (0.78, 0.90)
PPV 0.46 (0.31, 0.63) 0.43 (0.28, 0.59)
Initially, 222 patients underwent the methylation assay, and NPV 0.80 (0.74, 0.86) 0.80 (0.73, 0.85)
13 of them (6% of the total cohort) were withdrawn from Positive DLR 2.61 (1.54, 4.42) 2.26 (1.34, 3.83)
further analysis owing to inadequate β-actin amplification (9 Negative DLR 0.74 (0.59, 0.92) 0.77 (0.63, 0.95)
from DUMC and 4 from EVMS). Hence, a total of 209 patients DLR ¼ diagnostic likelihood ratio; NPV ¼ negative predictive value;
were evaluated in this study with 111 from DUMC and 98 from PPV ¼ positive predictive value.
387.e4 J.J. Fantony et al. / Urologic Oncology: Seminars and Original Investigations 33 (2015) 387.e1–387.e6

Table 3
Performance of methylation assay categorized by bladder cancer history: using “believe-the-positive” and Duke thresholds

Subgroup Hematuria Prior NMIBC Prior BCG

Bladder cancer prevalence

Apparent 0.29 (0.19–0.42) 0.41 (0.33–0.49) 0.41 (0.29–0.54)
True 0.25 (0.15–0.37) 0.25 (0.18–0.33) 0.27 (0.17–0.40)
Sensitivity 0.75 (0.48–0.93) 0.50 (0.33–0.67) 0.41 (0.18–0.67)
Specificity 0.86 (0.73–0.94) 0.63 (0.53–0.72) 0.59 (0.43–0.73)
PPV 0.63 (0.38–0.84) 0.62 (0.52–0.71) 0.27 (0.12–0.48)
NPV 0.91 (0.79–0.98) 0.79 (0.69–0.87) 0.73 (0.56–0.86)
Positive DLR 5.25 (2.50–11.02) 1.32 (0.88–1.98) 1.00 (0.51–1.94)
Negative DLR 0.29 (0.12–0.69) 0.81 (0.56–1.15) 1.00 (0.63–1.60)

DLR ¼ diagnostic likelihood ratio; NPV ¼ negative predictive value; PPV ¼ positive predictive value.

The diagnostic performance of the urine assay in several
important patient subgroups was assessed using the believe-
the-positive and the Abern et al. thresholds as shown in
Table 3. Interestingly, the methylation assay performed
better in subjects with hematuria than in subjects undergoing
NMIBC surveillance, a difference that has important clinical
ramifications and may be related to spectrum effects. Of
note, there were a larger proportion of patients in the
hematuria cohort that had high-grade or muscle invasive
bladder cancer at biopsy, but this was not statistically
significant (P ¼ 0.197). The proportion of current or former
smokers was higher in the NMIBC (15% and 60%,
respectively) cohort than the hematuria cohort (9% and
48%, respectively, P o 0.001). Additionally, the assay had
slightly reduced accuracy for patients with prior BCG
therapy as compared with the overall bladder cancer
surveillance subgroup. As prior studies have demonstrated
a change in the methylation of several genes in cigarette
smokers [20], the diagnostic performance of the methylation
assay stratified by smoking status is also shown in Table 4.
The test performed better in smokers than nonsmokers, an
interesting finding that requires validation in other cohorts.
suspicious cystoscopies were classified as negative. As a
sensitivity analysis, suspicious cystoscopies were reclassi-
fied as positive and the data reanalyzed. In this analysis the
AUC for TWIST1 changed from 0.667 (95% CI: 0.582–
0.752) to 0.646 (95% CI: 0.562–0.730) and the AUC for
NID2 changed from 0.637 (95% CI: 0.542–0.732) to 0.643
(95% CI: 0.557–0.729), differences that were not statisti-
cally or clinically significant. These results suggest that the
diagnostic performances of TWIST1 and NID2 are inde-
pendent of the classification of suspicious cystoscopies as
either positive or negative.
The ROC curve for the urine test consisting of a
combination of both TWIST1 and NID2 is shown in
Fig. 2. The AUC for the combined model was 0.656
(95% CI: 0.555–0.751), which was not statistically different
from that of either TWIST1 or NID2 alone, indicating that
these 2 urine methylation markers are unlikely to be
complementary in diagnosing bladder cancer.
3.3. Clinical utility

The decision curve for the methylation assay is shown in

3.2. ROC curves Fig. 3. At thresholds greater than 10%, the methylation
assay model provided only modest net benefit over perform-
ROC curves for the individual urine methylation markers ing cystoscopy on all patients with suspicion for bladder
are shown in Fig. 1. To generate these ROC curves, tumor (i.e., for hematuria or history of bladder cancer).

Table 4
Performance of methylation assay categorized by cigarette smoking history: using “believe-the-positive” and Duke thresholds

Subgroup Current smoker Former smoker Never smoker

Bladder cancer prevalence

Apparent 0.57 (0.34–0.77) 0.34 (0.25–0.44) 0.25 (0.15–0.39)
True 0.39 (0.20–0.61) 0.28 (0.19–0.38) 0.16 (0.78–0.29)
Sensitivity 0.78 (0.40–0.97) 0.56 (0.35–0.75) 0.44 (0.14–0.79)
Specificity 0.57 (0.29–0.82) 0.74 (0.62–0.84) 0.78 (0.64–0.89)
PPV 0.54 (0.25–0.81) 0.45 (0.28–0.64) 0.29 (0.08–0.58)
NPV 0.80 (0.44–0.97) 0.81 (0.70–0.90) 0.88 (0.74–0.96)
Positive DLR 1.81 (0.90–3.65) 2.16 (1.28–3.64) 2.04 (0.82–5.10)
Negative DLR 0.39 (0.11–1.43) 0.60 (0.38–0.93) 0.71 (0.39–1.30)

DLR ¼ diagnostic likelihood ratio; NPV ¼ negative predictive value; PPV ¼ positive predictive value.
 J.J. Fantony et al. / Urologic Oncology: Seminars and Original Investigations 33 (2015) 387.e1–387.e6 387.e5

1.0 multi-institutional level. Although the differences in diag-

nostic performance between the Abern and Renard thresh-
0.8
olds were not statistically significant, they were clinically
significant, and with a larger sample size they may have
been statistically apparent.
0.6
Sensitivity

The diagnostic performances outlined in Tables 2 and 3

0.4
0.2
TWIST (AUC=0.67)
NID2 (AUC=0.64)
0.0
1.0 0.8 0.6 0.4
Specificity
Fig. 1. Receiver operating characteristics curve for DNA methylation assay
for TWIST1 (solid) and NID2 (dotted) individually (log transformed).
4. Discussion
In this study we expand our previously published single-
institution cohort of 111 patients with an additional 98
patients from a second academic medical center and were
again unable to replicate the promising results for methy-
lated urine TWIST1 and NID2 of Renard et al. [8,9]. When
compared to our previous study, none of the diagnostic
metrics improved to a statistically significant degree despite
the additional patients, irrespective of the threshold used to
define test positivity. We again showed that thresholds for a
positive test might not always be transferable from the
developmental stage to a “real-life” clinical scenario, and
that validation of these thresholds should be performed on a
1.0
TWIST1 + NID2
(AUC = 0.657)
as well as in Figs. 1 and 2 show that the methylation of
urinary TWIST1 and NID2, viewed as either individual
tests or combined, is not adequate for clinical use. The
decision curve analysis shown in Fig. 3 shows that only at
high thresholds of missed bladder cancer diagnoses does the
methylation assay provide any clinical utility, and even then
it is modest. Although the exact morbidity load of office-
flexible cystoscopy is unknown, most would agree that it is
quite low. Therefore, most clinicians would have a very low
0.2 0.0 tolerance for missing a bladder cancer diagnosis by fore-
going cystoscopy (in other words, the threshold value
would be extremely low). As seen in the figure, the
methylation assay provides no net benefit at clinically
relevant thresholds.
Cigarette smoking is well known to have an effect on
DNA methylation in bladder cancer. Although Yegin et al.
[21] reported no differences in methylation of TWIST1 and
NID2 between smokers and nonsmokers, this study was
likely underpowered. In our cohort the assay was more
sensitive in current smokers (78%) compared with former
smokers (56%) and nonsmokers (44%). This may be due to
our subgroup of current smokers being small (23 patients in
total) and tending to have more abnormal cystoscopies. It
should be noted that the smoking status of the included
patients was obtained retrospectively, and there was a
moderate number of subjects whose smoking status was
unknown. The difference in the sensitivity should be
viewed through this limitation. However, there may be
value in exploring this assay as a screening test for subjects
who are active smokers. Furthermore, the increasing sensi-
tivity of the assay from nonsmokers to former and current
1.0

0.8 None

All
0.8

TWIST1+NID2
0.6
Sensitivity

0.6
Net benefit

0.4
0.4
0.2

0.2
0.0

0.0

0 20 40 60 80 100
Threshold probability (%)
1.0 0.8 0.6 0.4 0.2 0.0
Specificity Fig. 3. Decision curve analysis showing the net benefit of not performing
cystoscopy on anyone (solid black line), all patients (gray line), and the risk
Fig. 2. Receiver operating characteristics curve for the combined mode of score logistic regression model combining TWIST1 and NID2
TWIST1 and NID2 (log transformed). (dashed line).
387.e6 J.J. Fantony et al. / Urologic Oncology: Seminars and Original Investigations 33 (2015) 387.e1–387.e6
smokers raises the speculation of a progressive methylation
of these genes over time and whether this is reversible.
TWIST1 and NID2 displayed evidence of spectrum
effect in that test performance varied by patient subgroups
[22]. For instance, we noted that the combined TWIST1/
NID2 test performed better in subjects undergoing hema-
turia evaluation than in subjects undergoing bladder cancer
surveillance. This finding is important and highlights some
pitfalls that may occur when developing a diagnostic or
screening test for large-scale clinical use. In our cohort, the
hematuria patients had a higher proportion of high-grade
and muscle invasive pathology at biopsy. Although it was
not statistically significant owing to the small number of
patients, it is an important finding as this subgroup is similar
to the developmental cohort analyzed in the Renard paper.
Differences such as pretest probability of bladder cancer
and baseline patient characteristics between the develop-
mental cohorts and the general population make multi-
institutional external validation necessary to determine the
true applicability of a diagnostic test, such as this assay.
5. Conclusions
There remains a significant need to reduce cost and improve
the diagnostic management of patients diagnosed with bladder
cancer as well as those undergoing an initial hematuria workup.
The DNA methylation assay of TWIST1 and NID2 is not
superior to urine tests currently in clinical use in this validation
study at 2 separate academic medical centers. For future
direction, further evaluation of this test in smokers is warranted.
References
[1] Miremami J, Kyprianou N. The promise of novel molecular markers
in bladder cancer. Int J Mol Sci 2014;15:23897–908.
[2] Shariat SF, Karam JA, Lotan Y, Karakiewizc PI, et al. Critical
evaluation of urinary markers for bladder cancer detection and
monitoring. Rev Urol 2008;10:120–35.
[3] van der Aa MN, Steyerberg EW, Sen EF, Zwarthoff EC, Kirkels WJ,
van der Kwast TH, et al. Patients' perceived burden of cystoscopic
and urinary surveillance of bladder cancer: a randomized comparison.
BJU Int 2008;101:1106–10.
[4] Lokeshwar VB, Habuchi T, Grossman HB, Murphy WM, Hautmann
SH, Hemstreet GP 3rd, et al. Bladder tumor markers beyond
cytology: International Consensus Panel on bladder tumor markers.
Urology 2005;66(6 Suppl. 1):35–63.
[5] Yafi FA, Brimo F, Steinberg J, Aprikian AG, Tanguay S, Kassouf W,
et al. Prospective analysis of sensitivity and specificity of urinary
cytology and other urinary biomarkers for bladder cancer. Urol Oncol
2015;33:66.e25-31.
[6] Fridrichova I, Smolkova B, Kajabova V, Zmetakova I, Krivulcik T,
Mego M, et al. CXCL12 and ADAM23 hypermethylation are
associated with advanced breast cancers. Transl Res 2015;165:717–30.
[7] Ng JM, Yu J. Promoter hypermethylation of tumour suppressor genes
as potential biomarkers in colorectal cancer. Int J Mol Sci 2015;16:
2472–96.
[8] Renard I, Joniau S, van Cleynenbreugel B, Collette C, Naome C,
Vlassenbroeck I, et al. Identification and validation of the methylated
TWIST1 and NID2 genes through real-time methylation-specific
polymerase chain reaction assays for the noninvasive detection of
primary bladder cancer in urine samples. Eur Urol 2010;58:96–104.
[9] Abern MR, Owusu R, Inman BA. Clinical performance and utility of
a DNA methylation urine test for bladder cancer. Urol Oncol 2014;
32:51:e21-6.
[10] Li K, Lin T, Fan X, Duan Y, Huang J, et al. Diagnosis of narrow-band
imaging in non-muscle-invasive bladder cancer: a systematic review and
meta-analysis. Int J Urol 2013;20:602–9.
[11] Osborne JW. Improving your data transformations: applying the
Box-Cox transformation. Pract Asses Res Eval 2010;15:1–9.
[12] DeLong ER, DeLong DM, Clarke-Pearson DL. Comparing the areas
under two or more correlated receiver operating characteristic curves:
a nonparametric approach. Biometrics 1988;44:837–45.
[13] Fawcett T. An introduction to ROC analysis. Pattern Recogn Lett 2006;
27:861–74.
[14] Marshall RJ. The predictive value of simple rules for combining two
diagnostic tests. Biometrics 1989;45:1213–22.
[15] McIntosh MW, Pepe MS. Combining several screening tests:
optimality of the risk score. Biometrics 2002;58:657–64.
[16] Pepe MS. The statistical evaluation of medical tests for classification
and prediction. New York: Oxford University Press, 2003.
[17] Vickers AJ, Elkin EB. Decision curve analysis: a novel method for
evaluating prediction models. Med Decis Making 2006;26:565–74.
[18] Robin X, Turck N, Hainard A, Tiberti N, Lisacek F, Sanchez JC, et al.
pROC: an open-source package for R and Sþ to analyze and compare
ROC curves. BMC Bioinformatics 2011;12:77.
[19] R Development Core Team. R: a language and environment for
statistical computing. 2011, R Foundation for Statistical Computing:
Vienna.
[20] Besaratinia A, Cockburn M, Tommasi S. Alterations of DNA
methylome in human bladder cancer. Epigenetics 2013;8:1013–22.
[21] Yegin Z, Gunes S, Buyukalpelli R. Hypermethylation of TWIST1 and
NID2 in tumor tissues and voided urine in urinary bladder cancer
patients. DNA Cell Biol 2013;32:386–92.
[22] Mulherin SA, Miller WC. Spectrum bias or spectrum effect? Subgroup
variation in diagnostic test evaluation. Ann Intern Med 2002;
137:598–602.