ORIGINAL ARTICLE

Diagnostic outcomes of esophageal cancer by artificial

intelligence using convolutional neural networks
Yoshimasa Horie, MD,1,2 Toshiyuki Yoshio, MD,1,3 Kazuharu Aoyama, MM,4 Shoichi Yoshimizu, MD,1
Yusuke Horiuchi, MD,1 Akiyoshi Ishiyama, MD,1 Toshiaki Hirasawa, MD,1,3 Tomohiro Tsuchida, MD,1
Tsuyoshi Ozawa, MD,3,5 Soichiro Ishihara, MD,3,5 Youichi Kumagai, MD,6 Mitsuhiro Fujishiro, MD,7
Iruru Maetani, MD,2 Junko Fujisaki, MD,1 Tomohiro Tada, MD3,4,8
Tokyo, Saitama, Japan

Background and Aims: The prognosis of esophageal cancer is relatively poor. Patients are usually diagnosed at
an advanced stage when it is often too late for effective treatment. Recently, artificial intelligence (AI) using deep
learning has made remarkable progress in medicine. However, there are no reports on its application for diag-
nosing esophageal cancer. Here, we demonstrate the diagnostic ability of AI to detect esophageal cancer
including squamous cell carcinoma and adenocarcinoma.
Methods: We retrospectively collected 8428 training images of esophageal cancer from 384 patients at the Cancer
Institute Hospital, Japan. Using these, we developed deep learning through convolutional neural networks
(CNNs). We also prepared 1118 test images for 47 patients with 49 esophageal cancers and 50 patients without
esophageal cancer to evaluate the diagnostic accuracy.
Results: The CNN took 27 seconds to analyze 1118 test images and correctly detected esophageal cancer cases with a
sensitivity of 98%. CNN could detect all 7 small cancer lesions less than 10 mm in size. Although the positive predictive
value for each image was 40%, misdiagnosing shadows and normal structures led to a negative predictive value of 95%.
The CNN could distinguish superficial esophageal cancer from advanced cancer with an accuracy of 98%.
Conclusions: The constructed CNN system for detecting esophageal cancer can analyze stored endoscopic im-
ages in a short time with high sensitivity. However, more training would lead to higher diagnostic accuracy. This
system can facilitate early detection in practice, leading to a better prognosis in the near future. (Gastrointest
Endosc 2018;-:1-8.)

Esophageal cancer is the eighth most common cancer
worldwide, with an estimated 456,000 new cases each
year. It is the sixth most common cause of cancer-related
mortality with an estimated 400,000 deaths per year.1
Although the incidence of esophageal adenocarcinoma
(EAC) has been increasing globally, esophageal squamous
cell carcinoma (ESCC) remains the most common
histologic type of esophageal cancer in Asia and in Japan.
When esophageal cancer is diagnosed at an advanced
stage, it requires a highly invasive treatment, and its
prognosis is poor. Therefore, early detection is of great
importance.

Abbreviations: AI, artificial intelligence; CNN, convolutional neural
network; EAC, esophageal adenocarcinoma; ESCC, esophageal squa-
mous cell carcinoma; NBI, narrow-band imaging; PPV, positive predic-
tive value; NPV, negative predictive value; WLI, white-light imaging.
DISCLOSURE: All authors disclosed no financial relationships relevant
to this publication.
Copyright ª 2018 by the American Society for Gastrointestinal Endoscopy
0016-5107/$36.00
https://doi.org/10.1016/j.gie.2018.07.037
Received April 16, 2018. Accepted July 29, 2018.
Current affiliations: Department of Gastroenterology, Cancer Institute
Hospital, Japanese Foundation for Cancer Research, Tokyo, Japan (1),
Division of Gastroenterology and Hepatology, Department of Internal
Medicine, Toho University Ohashi Medical Center, Tokyo, Japan (2), Tada
Tomohiro Institute of Gastroenterology and Proctology, Saitama, Japan
(3), AI Medical Service Inc, Tokyo, Japan (4), Surgery Department, Sanno
Hospital, International University of Health and Welfare, Tokyo, Japan (5),
Department of Digestive Tract and General Surgery, Saitama Medical
Center, Saitama Medical University, Saitama, Japan (6), Department of
Gastroenterology, Graduate School of Medicine (7), Department of
Surgical Oncology, Graduate School of Medicine (8), The University of
Tokyo, Tokyo, Japan.
Reprint requests: Toshiyuki Yoshio, MD, PhD, Department of
Gastroenterology Cancer Institute Hospital, 3-8-31, Ariake, Koto-ku,
Tokyo 135-8550, Japan.

www.giejournal.org Volume -, No. - : 2018 GASTROINTESTINAL ENDOSCOPY 1

AI for diagnosing esophageal cancer
It is difficult to make an endoscopic diagnosis of ESCC
during the early stages with white-light imaging (WLI)
alone. Iodine staining has been used as the preferred
method of screening in high-risk patients. However, the
use of iodine staining is associated with problems, such
as chest pain or discomfort and an increased procedural
time. Narrow-band imaging (NBI) is a revolutionary tech-
nology of optical image-enhanced endoscopy that facili-
tates the detection of ESCC as a brownish area without
using iodine staining,2-4 and the results are available in a
moment by simply pressing a button. NBI is considered su-
perior to iodine for its ease of use and because it causes
less discomfort for patients. However, NBI has demon-
strated an insufficient sensitivity of 53% for detecting
ESCC when used by inexperienced endoscopists.5
Therefore, a real-time diagnosis supporting system may
be of great help for endoscopists in the detection of super-
ficial ESCC.
In recent years, artificial intelligence (AI) using deep
learning has made remarkable progress in various medical
fields, especially as a system to screen medical images. These
fields include diagnosis in radiation oncology,6 skin cancer
classification,7 diabetic retinopathy,8 histologic classification
of gastric biopsy specimens,9 and characterization of
colorectal lesions using endocytoscopy.10 We have also
reported on AI diagnoses using endoscopic images of
Helicobacter pylori infections11 and of gastric cancer
detection.12 However, no reports have assessed the
usefulness of AI in the diagnosis of esophageal cancer.
In the context of medical imaging, deep learning can
become a powerful supportive tool that can interpret med-
ical images based on an accumulated set of unique algo-
rithms. Deep learning allows computational models that
are composed of multiple processing layers to learn repre-
sentations of data with multiple levels of abstraction.13
To develop deep learning through a convolutional neu-
ral network (CNN) for the detection of esophageal cancer,
we constructed an AI-based diagnostic system that was
trained using a large number of EGD images. Here we
demonstrate the diagnostic ability of this system to detect
esophageal cancer including ESCC and EAC.
METHODS
Preparation of training and test image sets
For this single-center retrospective study, we obtained
EGD images from February 2016 to April 2017 from the
Cancer Institute Hospital, Tokyo, Japan, to develop an al-
gorithm for the detection of esophageal cancer. The
EGD was performed for screening or preoperative exami-
nations in daily clinical practice, and the images were
captured using high-resolution endoscopes (GIF-H290Z,
GIF-H290, GIF-XP290N, GIF-H260Z, GIF-H260; Olympus
Medical Systems, Co, Ltd, Tokyo, Japan) and standard
endoscope video systems (EVIS LUCERA CV-260/CLV-260,
2 GASTROINTESTINAL ENDOSCOPY Volume -, No. - : 2018
Horie et al
TABLE 1. Patient and lesion characteristics in the test image sets
Patient characteristics (n [ 47) Value
Sex, male/female 41/6
Median age, y (range) 70 (48-81)
Lesion characteristics (n [ 49)
Median tumor size, mm (range) 20 (5-700)
Tumor location, Ce/Ut/Mt/Lt/Ae 0/8/23/10/8
Macroscopic type
Superficial cancer, 0-I/0-IIa/0-IIb/0-IIc 1/6/13/23
Advanced cancer 1/2/3/4 0/3/3/0
Depth of tumor, T1a/T1b/T2-4 40/2/7
Histopathology, ESCC/EAC 41/8
ESCC, Esophageal squamous cell carcinoma; EAC, esophageal adenocarcinoma; Ut,
upper thoracic esophagus; Mt, middle thoracic esophagus; Lt, lower thoracic
esophagus; Ae, abdominal esophagus; Ce, cervical esophagus.
EVIS LUCERA ELITE CV-290/CLV-290SL; Olympus Medical
System).
We collected 8428 training images of esophageal cancer
lesions that were histologically proven to be SCC or adeno-
carcinoma in 384 patients. The training esophageal cancer
images included 397 lesions of ESCC that consisted of 332
lesions of superficial cancer and 65 lesions of advanced
cancer. A total of 32 images of EAC were also included
for training and consisted of 19 lesions of superficial can-
cers and 13 lesions of advanced cancers. Multiple images
were prepared for each lesion by controlling the air vol-
ume in the esophagus to distend the esophageal wall
and control the distance to the lesions. The training images
included standard WLI and NBI, excluding poor quality im-
ages resulting from halation, blur, defocus, mucus, and
poor insufflation of the air. Magnified images by magni-
fying endoscopy were also excluded from this study. All im-
ages of esophageal cancer lesions were marked manually
by 1 author (Y.H.) who is a well-experienced endoscopist,
with more than 5 years of experience and having per-
formed 3000 examinations in a high-volume cancer center.
To evaluate the diagnostic accuracy of the constructed
CNN, an independent set of test images were prepared
from 47 patients with 49 esophageal cancer lesions,
including 41 ESCCs and 8 EACs (162 images of esophageal
cancer and 376 images without cancer) and from 50 patients
with no esophageal cancer (573 images of the noncancerous
part of the esophagus). Of the 47 patients with esophageal
cancer lesions, 45 had 1 lesion each, whereas 2 of them
had 2 lesions each. There was a median number of 2 images
(range, 1-6) for every lesion with WLI and 1 image per lesion
(range, 0-4) with NBI. During the procedure, the entire
esophagus was observed, and images of every part of the
esophagus were captured using WLI and NBI.
All cases of esophageal cancer were confirmed to have
no other cancer using WLI, NBI, iodine staining, and
follow-up endoscopy after the treatment.
www.giejournal.org
Horie et al AI for diagnosing esophageal cancer

Figure 1. Examples of AI diagnosed images. A and B, For WLI and NBI, the CNN recognized esophageal cancer (white square) and was matched with a
green square that we used to identify the cancer. C and D, The CNN could not recognize esophageal cancer (green square) in WLI but could recognize it
in NBI (white and green squares), and it was matched with the correct diagnosis (green square). All recognized lesions were diagnosed as superficial
cancers. WLI, White-light imaging; NBI, narrow-band imaging; CNN, convolutional neural networks; AI, artificial intelligence.

Constructing a CNN algorithm
To construct an AI-based diagnostic system, we used a
deep neural network architecture called Single Shot Multi-
Box Detector (https://arxiv.org/abs/1512.02325), without
altering its algorithm as described previously.12 The
Single Shot MultiBox Detector is a deep CNN that
consists of 16 layers or more. A Caffe deep learning
framework, one of the most widely used frameworks
originally developed at the Berkeley Vision and Learning
Center, was used to train, validate, and test the CNN.
All layers of the CNN were fine-tuned using stochastic
gradient descent with a global learning rate of .0001.
Each image was resized to 300
300 pixels; the bounding
box was also resized accordingly. These values were set up
by trial and error to ensure all data were compatible with
Single Shot MultiBox Detector.
Outcome measures of AI diagnosis
After constructing the CNN using the training image set,
performance evaluation was performed using independent
test images. When the trained CNN detected esophageal
cancer from the input data of the test images, a disease
name (superficial or advanced esophageal cancer) was as-
signed and a rectangular frame was displayed in the endo-
scopic image so as to surround the lesion of interest.
Several criteria were chosen to evaluate the diagnostic per-
formance of the CNN for the detection of esophageal cancer
and included the following. When the CNN could recognize
even a part of the cancer, the CNN was considered as correct,
because it is sometimes difficult to identify the whole bound-
ary of cancer in 1 image. However, even when there was can-
cer in the frame judged by the CNN, it was considered
incorrect if it contained wide noncancerous sites occupying
more than 80% of the frame. In the image with findings
that indicated cancer, when the CNN recognized noncan-
cerous sites as cancerous, it was judged as a false-positive
result. When the CNN diagnosed 2 noncancerous sites as can-
cers in 1 image without cancer, it was counted as 1 false pos-
itive. In the image without cancer, when the CNN recognized
a noncancerous site as cancer, it was considered a false pos-
itive. In the case with 2 lesions, only when the CNN could
recognize both lesions was it considered a correct diagnosis.

www.giejournal.org Volume -, No. - : 2018 GASTROINTESTINAL ENDOSCOPY 3

AI for diagnosing esophageal cancer Horie et al

TABLE 2. Detailed results of AI diagnosis for each case P <.05

Results of biopsy 100

specimen
Cancer* Noncancer
80

AI diagnosis by WLI 60

Sensitivity
Cancer 38 35
40
Noncancer 9 15
AI diagnosis by NBI* 20
Cancer 41 28
Noncancer 5 22 0
Comprehensive AI diagnosisy A WLI NBI Comprehensive
diagnosis
Cancer 46 42
100
Noncancer 1 8 P = .32
AI, Artificial intelligence; WLI, white-light imaging; NBI, narrow-band imaging. 80
*Because there was 1 case without NBI images, a total number of cases diagnosed by

Sensitivity
NBI were 46 cases.
yIn the comprehensive CNN diagnosis, when diagnosed with either WLI or NBI, we
60
defined the CNN had been correctly diagnosed.
40

Definition of the analysis 20

The sensitivity for the CNN’s ability to detect esophageal
cancer for each case was calculated as follows: number of cases
in which the CNN correctly diagnosed esophageal cancer O
all cases of esophageal cancer. In a comprehensive analysis,
when the CNN detected the esophageal cancer with either
WLI or NBI, we defined this as the correct diagnosis.
Sensitivity, specificity, positive predictive value (PPV),
and negative predictive value (NPV) for the CNN’s ability
to detect esophageal cancer for each image was calculated
as follows: sensitivity, number of images in which the CNN
correctly diagnosed cancer O all images with cancer; spec-
ificity, number of images in which AI correctly diagnosed
noncancerous lesions O all images without cancer; PPV,
number of images in which the CNN correctly diagnosed
cancer O all images in which the CNN diagnosed cancer;
and NPV, number of images in which the CNN correctly
diagnosed noncancerous lesions O all images the CNN
diagnosed as noncancerous lesions. Macroscopic types of
lesions were assessed according to the Japanese Classifica-
tion of Esophageal Cancer edited by the Japan Esophageal
Society.14 Superficial cancer included mucosal (T1a) and
submucosal (T1b) cancer, and advanced cancer included
any cancer that was deeper than the muscularis propria
(T2-4).14
Statistical analysis
All continuous variables are expressed as the median
with a range. Statistical analyses were conducted using
Fisher’s exact test using the GraphPad Prism (GraphPad
Software, Inc, La Jolla, Calif). A P < .05 was considered sta-
tistically significant. To compare the sensitivities of WLI,
NBI, and a comprehensive diagnosis for each case, a P
value was adjusted by Bonferroni correction.
0
B WLI NBI
Figure 2. The sensitivity of the AI diagnosis for each case and for each
image. A, The sensitivity of the CNN diagnosis for each case is shown in
the graph. The sensitivity for the comprehensive diagnosis was signifi-
cantly higher than that in the WLI (P < .05). B, The sensitivity of the
CNN diagnosis for each image in NBI was slightly higher than in WLI
but not significantly. AI, Artificial intelligence; WLI, White-light imaging;
NBI, narrow-band imaging; CNN, convolutional neural networks.
Ethics
This study was approved by the Institutional Review
Board of the Cancer Institute Hospital (no. 2016-1171)
and the Japan Medical Association (ID JMA-II A00283).
RESULTS
Characteristics of patients and lesions in test
image sets
The CNN required 27 seconds to analyze the test image
sets of 1118 images for 97 patients. The median tumor size
was 20 mm, and 82% of the lesions were mucosal cancers
(T1a), 4% were submucosal cancers (T1b), and 14% were
advanced cancers deeper than the muscularis propria
(T2-4). For histopathology, 84% of the lesions were SCC
and 16% were adenocarcinoma (Table 1).
The example test images are shown in Figure 1. When
the CNN recognized a cancer, the CNN marked the
cancer with a white square that indicated superficial and
advanced cancers (Fig. 1A, B, and D). We marked the
cancer lesion by a green square independently (Fig. 1A-
D). By comparing the green square with the white
square, we were able to identify if the CNN could

4 GASTROINTESTINAL ENDOSCOPY Volume -, No. - : 2018 www.giejournal.org

Horie et al AI for diagnosing esophageal cancer

TABLE 3. Detailed results of the CNN diagnosis for each image TABLE 5. Causes for false positives and false negatives in the CNN
diagnosis
Results of biopsy specimen
Cancer Noncancer No. of images
Causes for false positives (%)
CNN diagnosis by WLI
Shadow 95 (50)
Cancer 61 109
Normal structure 61 (32)
Noncancer 24 364 EGJ/left main bronchus/vertebral body) 29/25/7
CNN diagnosis by NBI Benign lesion 32 (17)
Cancer 64 87 Post-ER scar/focal atrophy/Barrett’s esophagus/ (13/9/5/4/1/1/1)
inflammation/intraepithelial neoplasia/
Noncancer 13 396
keratinization /ectopic gastric mucosa
CNN, convolutional neural networks; WLI, white-light imaging; NBI, narrow-band
imaging.
Causes for false negatives
Inflammation in background mucosa 10 (25)

TABLE 4. Accuracy in diagnosing superficial versus advanced cancer
WLI NBI All images
Superficial cancer, % 100 (75/75) 99 (67/68) 99 (142/143)
Advanced cancer, % 100 (14/14) 82 (9/11) 92 (23/25)
All cancer, % 100 (89/89) 96 (76/79) 98 (165/168)
Accuracy: number of superficial or advanced cancer images that were diagnosed
correctly by the CNN O number of images that the CNN diagnosed as cancer.
WLI, White-light imaging; NBI, narrow-band imaging; CNN, convolutional neural
networks.
diagnose the esophageal cancer or not. In the first
example, the CNN could diagnose the cancer in both the
WLI and the NBI (Fig. 1A and B). In the second example,
the CNN could not diagnose cancer in the WLI but could
diagnose the cancer in the NBI (Fig. 1C and D). Both
lesions in the example were diagnosed as superficial
cancer (Fig. 1A, B, and D).
CNN diagnosis for each case
The CNN correctly diagnosed 46 of 47 cases of esopha-
geal cancers with the comprehensive diagnose of WLI and
NBI (Table 2). Notably, CNN could detect all 7 lesions less
than 10 mm in size.
The sensitivity of the comprehensive CNN diagnosis was
98%, whereas the sensitivity of the CNN diagnosis based on
WLI and on NBI was 81% and 89%, respectively (Fig. 2A).
The NBI was more sensitive to diagnosing esophageal
cancer compared with WLI, although no significant
difference was noted (Fig. 2A). The comprehensive CNN
diagnosis of WLI and NBI had significantly higher
sensitivity than did those with WLI only (Fig. 2A). The
sensitivity of ESCC for WLI, NBI, and the comprehensive
CNN diagnosis was 79%, 89%, and 97%, respectively. The
sensitivity of EAC for WLI, NBI, and the comprehensive
CNN diagnosis was 88%, 88%, and 100%.
CNN diagnosis for each image
Detailed results of the CNN diagnosis for each image are
shown in Table 3. The sensitivity of the CNN diagnosis
including WLI and NBI was 77%, correctly detecting 125
cancer lesions of all 162 images with cancer (Table 3).
Obscure ESCC in WLI 7 (17)
Barrett’s esophageal EAC 4 (10)
Difficult conditions (lesions were too distant, only 20 (49)
piece of lesion)
CNN, Convolutional neural networks; WLI, white-light imaging; ESCC, esophageal
squamous cell carcinoma; EAC, esophageal adenocarcinomas; ER, endoscopic
resection; EGJ, esophagogastric junction.
The CNN diagnosis of the images also had a specificity of
79%, PPV of 39%, and NPV of 95%. Although not
significant, NBI tended to have higher sensitivity than
WLI (83% vs 72%, P Z .32) (Fig. 2B). The sensitivity of
the CNN diagnosis for ESCC on WLI and NBI was 72%
and 86%, respectively, and that for EAC was 69% and
71%, respectively.
CNN diagnosis: superficial versus advanced
cancer
Each image of esophageal cancer, whether taken with
WLI or NBI, was separately diagnosed as either superficial
or advanced cancer by the CNN. The accuracy in diagnosis
was quite high for both imaging modalities. The diagnostic
accuracy was 99% (142/143) for superficial cancer and 92%
(23/25) for advanced cancer (Table 4). The accuracy in
diagnosis for ESCC and EAC was 99% (146/147) and 90%
(19/21), respectively.
Causes for false positives and false negatives
The causes for false positives and negatives in CNN di-
agnoses are summarized in Table 5 in order of
frequency. The most-frequent cause of a false positive
was a shadow (Fig. 3A) accounting for 50% of the false
positives. Several normal structures also caused false
positives including the esophagogastric junction
(Fig. 3B), the left main bronchus (Fig. 3C), and a
vertebral body (Fig. 3D). Some benign lesions were also
sometimes misdiagnosed as cancer such as
postendoscopic resection scars (Fig. 3E) and focal
atrophy (Fig. 3F).
Nearly half of the false-negative images were because of
difficult conditions such as the lesions being too distant
(Fig. 4A) or the presence of just a part of the lesion

www.giejournal.org Volume -, No. - : 2018 GASTROINTESTINAL ENDOSCOPY 5

AI for diagnosing esophageal cancer
Figure 3. Examples of false-positive images. The white squares here indicate areas that were misdiagnosed as cancer. A, Shadow. B, Esophagogastric
junction. C, Left main bronchus. D, Vertebral body. E, Post–endoscopic resection scar. F, Focal atrophy.
(Fig. 4B). The other common cause for a false negative was
esophageal inflammation in the background mucosa
(Fig. 4C). The CNN also missed some obscure ESCC
lesions, particularly with the WLI (Fig. 4D), which were
sometimes difficult to diagnose even by expert
endoscopists. Four lesions of EAC were missed; this
included an obvious lesion (Fig. 4E). This may be
because educational images for EAC were not enough.
DISCUSSION
Here, we constructed an AI-based diagnostic system to
detect superficial and advanced esophageal cancer, using
a CNN trained by a large number of endoscopic images.
We prepared 8428 images of esophageal cancers that
were histologically proven to be SCC or adenocarcinoma
as training images. We demonstrated high diagnostic accu-
racy of the CNN by independent test images of 97 patients.
To the best of our knowledge, this is the first report that
evaluates the ability of the CNN to detect esophageal can-
cer in endoscopic images. We demonstrated that the sensi-
tivity of esophageal cancer diagnosis for each case by the
CNN was 98%. In addition, the CNN could detect all 7 le-
sions less than 10 mm in size.
To improve the performance of our CNN, we analyzed
the causes for false positives and false negatives. Half of
the false positives were caused by normal structures and
6 GASTROINTESTINAL ENDOSCOPY Volume -, No. - : 2018
Horie et al
benign lesions (Table 5). This can be corrected by deep
learning about each normal structure and benign lesion,
which will surely reduce false positives and improve the
PPV significantly. The other half of the false positives
were shadows (Table 5) in which dark areas in the
esophagus were sometimes misdiagnosed as cancer. It
may be difficult for the CNN to differentiate the shadow
from cancer, because a shadow does not have a
particular shape and changes in each situation. We need
to evaluate whether deeper learning can overcome this
limitation of the CNN diagnosis. It is true that the PPV of
the CNN diagnosis for each image was not high (39%)
(Table 3); however, the PPV of NBI with magnification
was reported to be 45% in experienced endoscopists and
35% in less-experienced endoscopists,5 which was not so
different from our outcomes. These results are important
to consider in terms of daily practice where false
positives are generally more acceptable than false
negatives. The CNN could not detect some esophageal
cancers with background mucosa inflammation, which
we sometimes encounter and require careful diagnoses.
To improve this system, we need to prepare training
images that include such cases more often.
In Japan, SCC remains the most common histologic
type of esophageal cancer. We therefore prepared more
training images of ESCC and included fewer images of
EAC. As a result, the sensitivity of the CNN diagnosis for
EAC was a little lower, especially with NBI than that for
www.giejournal.org
Horie et al
Figure 4. Examples of false-negative images. The following cancers were missed and estimated causes were as follows. A, Lesions were too distant. B,
Only a part of the lesion. C, Inflammation in the background mucosa. D, Obscure esophageal squamous cell carcinoma in white light imaging. E, Esoph-
ageal adenocarcinoma.
ESCC, although not significantly so. This can be improved
by preparing more training images of EAC.
One clinical application of the CNN is to check the
stored still images after examinations. We usually take 40
to 60 images for each screening EGD performed and store
them in a computer filing system. The CNN can check
these images in a short time and may identify some missed
cancer cases. Furthermore, the CNN can be applied to
detect esophageal cancers in the images of capsule endos-
copy. Another application is the real-time support for de-
tecting esophageal cancers. We are not sure if the CNN
can detect lesions that an expert endoscopist cannot
detect, because this system cannot visualize cancers
more clearly as done via NBI. However, we believe that
the CNN can support nonexperts in the detection of
esophageal cancer.5
It is difficult to discuss the additional effects of the CNN
in a retrospective study. Thus, we confirmed all referred
cases in the test image set and found that some lesions
were missed via EGD conducted in the previous hospital.
The test images included those for 23 patients who were
found to have esophageal cancer and were referred to
our hospital. In these patients, we detected 2 more lesions
that had been missed by EGD in the previous hospital;
CNN could also detect these 2 lesions. Moreover, CNN
could detect 2 more lesions that had been missed in
patients who were referred for the treatment of gastric
www.giejournal.org
AI for diagnosing esophageal cancer
cancer. These findings indicate the possibility that the
CNN can support nonexpert endoscopists to detect esoph-
ageal cancer.
To diagnose esophageal cancers that have been re-
corded on video, the CNN is required to analyze more
than 30 images in 1 second. However, having overcome
the technical limit, our present system can analyze 50 im-
ages in 1 second (.02 second for 1 image). Thus, we are
sure this system will work in medical videos, with the
same quality, which makes it possible to use the CNN dur-
ing procedures such as an EGD.
This study has several limitations. First, this was a single-
center retrospective study. However, we believe the results
are reliable because verification was made prospectively.
Second, we used only high-quality endoscopic images for
the training and test images. Thus, we are not sure if the
CNN can diagnose esophageal cancer using low-quality im-
ages, such as those with halation, with mucus, are blurry,
or are out of focus. Third, the number of EAC cases was
small. This is because we focused mainly on ESCC, which
is more common in Japan. We believe it is important to
accurately diagnose EAC as well, because the incidence is
increasing worldwide. Fourth, not all false positives were
proven to be noncancerous tissue by biopsy sampling.
However, we have confirmed there were no malignant le-
sions other than the target lesions by WLI, NBI, iodine
staining, and follow-up endoscopy after treatment in all
Volume -, No. - : 2018 GASTROINTESTINAL ENDOSCOPY 7
AI for diagnosing esophageal cancer
cases of esophageal cancer. Thus, we believe the diag-
nostic quality of the false positives.
In conclusion, AI-based diagnostic systems developed
by deep learning demonstrated high diagnostic accuracy
with high sensitivity to detect esophageal cancer with sur-
prising efficiency. For the first step, we analyzed only still
images, and we believe this system is useful for rechecking
the stored images after EGD. In the next study, we plan to
demonstrate the usefulness of the CNN system for medical
videos, which could help identify esophageal cancer during
EGD in real time. We hope the AI-based diagnostic system
will support us in detecting esophageal cancers and allow
early detection of esophageal cancer in daily clinical prac-
tice in the near future.
REFERENCES
1. GLOBOCAN. Estimated cancer incidence, mortality and prevalence
worldwide in 2012. International Agency for Research on Cancer–
World Health Organization. 2012. Available at: http://globocan.jarcfr/
Pages/fact_sheets_cancer.aspx. Accessed March 13, 2018.
2. Nagami Y, Tominaga K, Machida H, et al. Usefulness of non-magnifying
narrow-band imaging in screening of early esophageal squamous cell
carcinoma: a prospective comparative study using propensity score
matching. Am J Gastroenterol 2014;109:845-54.
3. Lee YC, Wang CP, Chen CC, et al. Transnasal endoscopy with narrow-
band imaging and Lugol staining to screen patients with head and
neck cancer whose condition limits oral intubation with standard
endoscope (with video). Gastrointest Endosc 2009;69:408-17.
8 GASTROINTESTINAL ENDOSCOPY Volume -, No. - : 2018
Horie et al
4. Kuraoka K, Hoshino E, Tsuchida T, et al. Early esophageal cancer can be
detected by screening endoscopy assisted with narrow-band imaging
(NBI). Hepatogastroenterology 2009;56:63-6.
5. Ishihara R, Takeuchi Y, Chatani R, et al. Prospective evaluation of
narrow-band imaging endoscopy for screening of esophageal
squamous mucosal high-grade neoplasia in experienced and less
experienced endoscopists. Dis Esoph 2010;23:480-6.
6. Bibault JE, Giraud P, Burgun A. Big data and machine learning in radi-
ation oncology: state of the art and future prospects. Cancer Lett
2016;382:110-7.
7. Esteva A, Kuprel B, Novoa RA, et al. Dermatologist-level classification of
skin cancer with deep neural networks. Nature 2017;542:115-8.
8. Gulshan V, Peng L, Coram M, et al. Development and validation of a
deep learning algorithm for detection of diabetic retinopathy in retinal
fundus photographs. JAMA 2016;316:2402-10.
9. Yoshida H, Shimazu T, Kiyuna T, et al. Automated histological classifi-
cation of whole-slide images of gastric biopsy specimens. Gastric Can-
cer 2018;21:249-57.
10. Misawa M, Kudo S, Mori Y, et al. Accuracy of computer-aided diagnosis
based on narrow-band imaging endocytoscopy for diagnosing
colorectal lesions: comparison with experts. Int J Comput Assist Radiol
Surg 2017;12:757-66.
11. Shichijo S, Nomura S, Aoyama K, et al. Application of convolutional
neural networks in the diagnosis of Helicobacter pylori infection based
on endoscopic images. EBioMed 2017;25:106-11.
12. Hirasawa T, Aoyama K, Tanimoto T, et al. Application of artificial
intelligence using a convolutional neural network for detecting gastric
cancer in endoscopic images. Gastric Cancer 2018 Jul;21(4):653-60.
13. LeCun Y, Bengio Y, Hinton G. Deep learning. Nature 2015;521:
436-44.
14. Kuwano H, Nishimura Y, Oyama T, et al. Guidelines for diagnosis and
treatment of carcinoma of the esophagus. April 2012 edited by the
Japan Esophageal Society. Esophagus 2015;12:1-30.
www.giejournal.org