See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/328983256

Using Octuplet Siamese Network For Osteoporosis Analysis On Dental

Panoramic Radiographs

Conference Paper  in  Conference proceedings: ... Annual International Conference of the IEEE Engineering in Medicine and Biology Society. IEEE Engineering in Medicine and Biology Society.
Conference · July 2018
DOI: 10.1109/EMBC.2018.8512755

CITATIONS READS
3 189

9 authors, including:

Vasileios Megalooikonomou Fan Yang

University of Patras Temple University
247 PUBLICATIONS   1,938 CITATIONS    20 PUBLICATIONS   319 CITATIONS   

SEE PROFILE SEE PROFILE

Xinyi Li Haibin Ling

Temple University Temple University
2 PUBLICATIONS   3 CITATIONS    143 PUBLICATIONS   6,856 CITATIONS   

SEE PROFILE SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Frailsafe View project

ARMOR project (FP7-ICT, No. 287720) View project

All content following this page was uploaded by Fan Yang on 19 November 2018.

The user has requested enhancement of the downloaded file.

 Using Octuplet Siamese Network For Osteoporosis Analysis On Dental
Panoramic Radiographs
Peng Chu1 , Chunjuan Bo2 , Xin Liang3 , Jie Yang4 , Vasileios Megalooikonomou5 ,
Fan Yang1 , Bingyao Huang1 , Xinyi Li1 , and Haibin Ling1?

Abstract— Dental Panoramic radiography (DPR) image pro-

vides a potentially inexpensive source to evaluate bone den-
sity change through visual clue analysis on trabecular bone 1
2
structure. However, dense overlapping of bone structures in
DPR image and scarcity of labeled samples make learning of 3 4
accurate mapping from DPR patches to osteoporosis condition
challenging. In this paper, we propose a deep Octuplet Siamese
Network (OSN) to learn and fuse discriminative features for
osteoporosis condition prediction using multiple DRP patches. 7
8
By exploring common features, OSN uses patches of eight 5 6
locations together to train the shared feature extractor. Feature
fusion for different location adopts both accumulation and con-
catenation with fully considering of patches’ spatial symmetry.
In our dedicated two-stage fine-tuning scheme, an augmented
texture analysis dataset is employed to prevent overfitting in
transferring weights learned on ImageNet to DPR dataset when
using merely 108 samples. Leave-one-out test shows that our
proposed OSN outperforms all other state of the art methods Fig. 1. DPR image with eight ROIs. Same color represents the same group.
in osteoporosis category classification task.

I. I NTRODUCTION
Osteoporosis is the most common bone disease affecting
millions of people every year. Reference-standard diagnosing
of osteoporosis conditions involves bone mineral density
(BMD) measurements by dual-energy X-ray absorptiometry
(DXA), which is expensive and has limited availability in
population. Recently, increasing works have demonstrated
the feasibility of using relative inexpensive panoramic radio-
graphy for BMD estimation. In [1], relation between DXA
measurements and some densitometric and linear measure-
ments including mandibular cortical thickness and panoramic
mandibular index are investigated. [2] and [3] verify the
correlations of several panoramic radiomorphometric indices
with lumbar spine and hip BMDs, and confirm the possibility
of using mental index, mandibular cortical index and visual
estimation of cortical as osteoporosis predictors. Image
feature information of trabecular bone provides another
promising source for osteoporosis evaluation. In [4], Parfitt et
al. show that texture changes in iliac trabecular bone, such
as surface texture, volume and thickness, can be used for
1
Peng Chu, Fan Yang, Bingyao Huang, Xinyi Li and Haibin
Ling are with Department of Computer and Information Sciences,
Temple University, Philadelphia, USA.
2
Chunjuan Bo is with College of Electromechanical Engineering,
Dalian Nationalities University, Dalian, China.
3
Xin Liang is with School of Stomatology, Dalian Medical
University, Dalian, China.
4 heavily on the huge amount of labeled data to find optimized
Jie Yang is with Division of Oral and Maxillofacial Radiology,
School of Dentistry, Temple University, Philadelphia, USA.
5
Vasileios Megalooikonomou is with Computer Engineering and
Informatics Department, University of Patras, Patras, Greece.
osteoporosis prediction. Faber et al. [5] exploit Fourier and
wavelet analysis to detect trabecular changes in osteoporosis.
Tosoni et al. [6] compare the pixel intensity values and fractal
dimensions in some selected mandibular regions. In [7] and
[8], correlation between various CBCT features and different
gender-age groups are investigated, which implicitly relate to
bone quality.
However, creating direct and reliable mapping from tra-
becular panoramic radiography images to osteoporosis con-
dition is challenging. Bo et al. [9] propose a pioneer work
to build a two stage SVMs learning framework to combine
information in multiple panoramic radiography patches for
osteoporosis status classification. However, only general pre-
defined feature descriptors are investigated in their work.
Each pixel in panoramic radiography image is the accu-
mulation of X-ray response in depth dimension, trabecular
bone response thus is usually overlapping with response
from other structures. Existing pre-defined feature descriptors
work well on modeling low level image clue. For remedy of
overlapping in panoramic radiography images, higher level
information is essential.
Recent deep learning methods, especially Convolutional
Neural Network (CNN) architecture, are reporting success
in learning those hierarchical features. Usually successful
training of a deep convolutional network (DCN) depends
value for millions of weights in DCN. Even for fine-tuning,
enough samples are still required to prevent the powerful pre-
trained model to overfit at noise in target dataset. However,

978-1-5386-3646-6/18/$31.00 ©2018 IEEE 2579


Patch Share all
augment weights
Feature
DPR image
extraction
Fig. 2. Overview of Octuplet Siamese Network.
big p small n is the most common type of problems in med-
ical image analysis. Specifically, in [9], each osteoporosis
condition prediction depends on eight panoramic radiography
patches while total number of labeled subjects is merely 40.
It is far insufficient for directly training or fine-tuning any
ordinary DCN.
Attacking these problems, in this paper we proposed an
Octuplet Siamese Network (OSN) trained through a two-
stage fine-tuning scheme to predict osteoporosis condition
using limited number of panoramic radiography images. We
carefully design our network architecture to fully explore
common features across eight input patches and use all
patches for feature learning. To further reduce complexity
and avoid overfitting, input features of single layer classifier
are accumulated based on their spatial symmetry in DPR
image before concatenation. During training process, we
design a dedicated two-stage fine-tuning scheme. Through
introducing an auxiliary texture dataset, we ensure the
smoothly transferring of model learned on ImageNet to
panoramic radiography dataset without losing its powerful
discrimination. By combining these strategies and training on
only 108 samples, our proposed OSN beats all other existing
method in classifying osteoporosis condition task.
II. METHODOLOGY
A. Problem Formulation
Dental panoramic radiography (DPR) is two-dimension
panoramic X-ray imaging of entire mouth, which is com-
monly used in dental diagnosis. Benefiting from the fact
that trabecular patterns are distributed in various places in
the oral cavity, DPR images thus contain rich information
of trabecular bone. Eight regions of interest (ROI) in DPR
images are further selected by dentist for relative better
imaging quality of trabecular bone structure. The eight ROIs
can be divided into four groups based on symmetry of human
face as illustrated in Figure 1.
Then our problem is formulated as using the eight patches
from each subject to predict osteoporosis condition of the
subject, e.g. osteoporosis or normal. For the n-th subject, it
2580
+
Addition
+
c Concatenation
+
Normal
c
OP
+
Linear
classifier
+
Feature
fusion
can be formally expressed as
ybn = G(In1 , In2 , . . . , In8 ), (1)
where ybn ∈ Y for Y = {yi }2i=1 is the two osteoporosis
categories, Inj ∈ RL×L for j = 1, 2, . . . , 8 is the j-th DPR
patch of pixel size L × L, and G(.) is the mapping function
to be learned.
B. Octuple Siamese Network
To efficiently fuse the visual clue from eight ROIs, we
designed a Siamese-style convolutional network with eight
subnets sharing all weights to do feature extraction and
classification jointly. Configuration of our proposed network
is shown in Figure 2. The whole network can be roughly
divided into two parts, deep feature extraction and classifi-
cation with fusion features.
In the first part, eight AlexNet [10] based deep CNNs
parallel encode patches in the eight ROIs into eight feature
vectors. As extracted from the same DPR image, each of the
eight patches shares similar low-level features. Thus, instead
of learning eight independent CNN feature extractors, we are
using a Siamese-style network which can be written as
Φnj = ϕj (Inj , WS , bS ), (2)
where Φnj ∈ RM is the M -dimension features output by the
j-th subnet, ϕj (.) is the j-th subnet, WS = {wl }Tl=1 and
bS = {bl }Tl=1 are all the weights of the T layers CNN.
Also, by sharing weights and updating mirrored in each
subnet, patches from all locations can be used together to
train one set of weights which increases the number of train-
ing samples by 8× and reduces the potential of overfitting.
Moreover, unlike other works to exploit features in FC6 or
even Pool5, we use complete AlexNet and take features
at FC8. This is based on two considerations. First, each
patch contains very limited context or layout information,
therefore spatial information encoded in feature from Pool5
or FC6 is useless in our application; Second, to use features
in Pool5 or FC6, either one more layer will be learned to
map the high dimension raw features to some desired low
dimension space, or increasing input dimension of classifier
 1×2
1×286
Fusion
AlexNet AlexNet AlexNet
Online data Online data
augmentation augmentation
ImageNet Texture DPR data
pre-train fine-tuning fine-tuning
Fig. 3. Two-stage fine-tuning.
is required. In both approaches, more weights without pre-
trained value are introduced, which increases the potential of
overfitting. Notice that, although here we are using AlexNet
based network for feature extraction, all other recent DCN
architectures will also be compatible in our setting.
We extend effective training samples for feature extraction
by sharing weights, whereas only the original limited number
of labels can be used for classifier training. To reduce the
complexity of classifier, besides choosing the simplest linear
classifier, we also minimize its input dimension. Instead
of directly concatenating, we first perform element-wise
addition between feature vectors whose patches belong to the
same group as shown in Figure 1. To further enhance this
invariance in feature, we add random horizontal flip during
data augmentation. By this change, we reduce half of the
weights in classifier and boost final classification accuracy by
2% compared with using direct concatenation. Finally, a fully
connected layer followed by softmax function is employed
as the classifier which outputs two possibilities, osteoporosis
(OP) and normal. The whole network can be written as
(i)
ybn∗ = argmax φsof tmax (wC Φn + bC )
i
Φn = [Φ1 (In1 ) + Φ2 (In2 ), ..., Φ7 (In7 ) + Φ8 (In8 )]
where Φn ∈ R4M are the concatenated features, [. . . ] is
vector concatenation, Φj (Inj ) is the feature vector associated
with patch Inj , wC and bC are the weights for the last fully
connected layer. Therefore, {WS , wC , bS , bC } are the set
of weights to be learned.
C. Texture Fine Tuning
Due to the scarcity of training samples, we design a two-
stage fine-tuning scheme to learn the weights as illustrated in
Figure 3. We start from the network pre-trained on ImageNet.
Images in ImageNet mostly are natural objects and captured
as colorful RGB images, whereas DPR patches contain
overlapped human bone structure X-ray response recorded as
grayscale images. Directly transferring a powerful network
learned on ImageNet using insufficient samples may cause
the network overfitting at noise and losing generalizability.
We use an auxiliary dataset to bring up the gap between the
two datasets.
Random
Ignoring the distracted response from other overlapping
initialization bone structure, trabecular bone structures are usually char-
Pre-trained acterized by the texture property of images. To capture any
possible type of texture, we create an augmented texture
dataset by merging several public texture analysis datasets.
In particular, ALOT [11], KTH-TIPS2 [12] and UIUC [13]
texture datasets are used. These datasets contain texture from
different sources which include both natural and manmade
patterns. All images in each dataset are first converted into
grayscale to match the style of DPR patches, then duplicated
to three channels and resized to adapt the input dimension
of AlexNet. Most images in texture dataset are not square.
Instead of directly resizing as in other classification task, we
crop images to squares to avoid interference from distortion
which is important for texture analysis. Finally, we create a
dataset with 55,000 images in 286 texture categories. When
fine-tuning on this dataset, we add a fully connected layer
with 286 output neurons on top of AlexNet. After the first
stage fine-tuning, the last fully connected layer is removed,
only weights in AlexNet are used to initialize the feature
extractor in OSN for the second stage fine-tuning on DPR
dataset.
Another key component in our two-stage fine-tuning is
online data augmentation. Commonly used offline data aug-
mentation generates augmented images from each real image
before training, which can only produce a fixed number of
augmented sets. Due to the limited number of real image
in DPR dataset, stochasticity in fixed augmented set will
still be exhausted after epochs. Online data augmentation, on
the other hand, produces augmented images real-time during
training, which needs additional computation but can gen-
erate unlimited combinations of augmentation operations as
training process continuing. We use online data augmentation
in both fine-tuning stages.
(i)
(3) III. EXPERIMENTS AND RESULTS
A. DPR Image
(4) We collect a dataset containing 108 images from 108
different subjects, which is an extension of the original 40
subjects reported in [9]. Our new dataset is relative balanced
and containing 52 subjects with osteoporosis and 56 normal
subjects. In each DPR image, dentist manually annotates the
eight ROIs with normalized pixel size of 50 × 50.
B. Environmental Settings
We use Caffe for network implementation. Caffe master
branch does not include an online augmentation layer, while
third party implementation cannot augment eight patches in-
dependently within each input patch tuple. We implemented
our own online data augmentation Python layer, where
standard random crop and horizontal flip were included.
For fine-tuning process, we downloaded AlexNet pre-
trained weights from Caffe Model Zoo. Newly added fully
connected layers were initialized by gaussian noise with
standard deviation of 0.01. We used fixed base learning rate
2581
 TABLE I
C LASSIFICATION RESULTS BY LEAVE - ONE - OUT.
Methods TSL[9] HARA SFTA OSN
NOPA(%) 67.86 67.86 71.43 75.00
OPA(%) 67.31 71.15 82.69 96.15
OA(%) 67.59 69.44 76.85 85.19
of 0.001, and new fully connected layers with 10 times of
base learning rate. Dropout with rate 0.5 was applied to all
internal fully connected layer to further reduce overfitting.
10K and 200 iterations were used in the two fine-tuning
stages.
C. Results And Discussion
In this part, we compare our proposed method with
methods in other well-established work and several shallow
methods where pre-defined features and general classifier are
adopted. TSL is the two-stage learning method reported in
[9]. Various feature descriptors are investigated in their work.
We compare the result of using HOG which achieved the best
performance in their original work.
We also compare our learning based feature with pre-
defined features. Haralick (HARA) feature [14] is widely
used in X-ray image analysis, which fuses multiple statistics
metric computed on gray-level co-occurrence matrix of input
image. Segmented Fractal Analysis of Textures (SFTA) [15]
is another type of texture analysis feature popular in medical
image analysis, which explores the fractal dimension of input
images sub-grayscale region. We use eight binary thresholds
for SFTA. For HARA and SFTA, Random Forest with 100
trees is employed as the classifier, and features from eight
ROIs are directly concatenated. For the proposed method, we
compare the results with or without using texture dataset for
fine-tuning (TF).
We adopt leave-one-out cross-validation as results eval-
uation criteria. To be specific, for n samples, there are n
different training and test sets. Thus, the final accuracy is cal-
culated by averaging the n classification outputs. In addition
to the overall accuracy (OA), we also report the accuracy on
osteoporosis subjects (OPA) and non-osteoporosis subjects
(NOPA).
As shown in Table. I, our proposed OSN outperforms other
shallow methods with large margins. Moreover, compared
with OSN+TF, OSN without texture database fine-tuning
suffers serious imbalanced accuracy in two osteoporosis cat-
egories, which results predictions with very poor precision.
Since training error is always balanced in both settings, fine-
tuning on a large and more comprehensive texture databased
is crucial for preventing OSN from overfitting at one cate-
gory.
IV. CONCLUSION
We presented a study of using DPR image for osteoporosis
condition evaluation and proposed an image-based osteo-
porosis classification method. The proposed method uses
Siamese-style convolutional network to learn and fuse image
View publication stats
features from multi-ROIs and a dedicated two-stage fine-
tuning scheme for training. In our experiments, proposed
method with two-stage fine-tuning achieves the best over-
OSN+TF all accuracy of 89.8%. Our results show that deep learn-
89.29
90.38
ing method with carefully designed network structure and
89.81 training scheme can be adopted for osteoporosis condition
analysis using DPR images, even for scarce training samples.
This result encourages future work using DPR image for
inexpensive osteoporosis prescreening.
ACKNOWLEDGMENT
This work is supported in part by US NSF Grants 1407156
and 1350521.
R EFERENCES
[1] K. Horner and H. Devlin, “The relationship between mandibular bone
mineral density and panoramic radiographic measurements,” Journal
of dentistry, vol. 26, no. 4, pp. 337–343, 1998.
[2] K. Z Vlasiadis, C. A Skouteris, G. A Velegrakis, I. Fragouli, J. M
Neratzoulakis, J. Damilakis, and E. E Koumantakis, “Mandibular ra-
diomorphometric measurements as indicators of possible osteoporosis
in postmenopausal women,” Maturitas, vol. 58, no. 3, pp. 226–235,
2007.
[3] A.F. Leite, P. T. de Souza Figueiredo, C. M. Guia, N. S. Melo, and
A. P. de Paula, “Correlations between seven panoramic radiomorpho-
metric indices and bone mineral density in postmenopausal women,”
Oral Surgery, Oral Medicine, Oral Pathology, Oral Radiology, and
Endodontology, vol. 109, no. 3, pp. 449–456, 2010.
[4] A. M. Parfitt, C. H. Mathews, A. R. Villanueva, M. Kleerekoper,
B. Frame, and D. S. Rao, “Relationships between surface, volume,
and thickness of iliac trabecular bone in aging and in osteoporosis.
implications for the microanatomic and cellular mechanisms of bone
loss.,” Journal of clinical investigation, vol. 72, no. 4, pp. 1396, 1983.
[5] T. D Faber, D. C Yoon, S. K Service, and S. C White, “Fourier and
wavelet analyses of dental radiographs detect trabecular changes in
osteoporosis,” Bone, vol. 35, no. 2, pp. 403–411, 2004.
[6] G. M Tosoni, A. G Lurie, A. E Cowan, and J. A Burleson, “Pixel in-
tensity and fractal analyses: detecting osteoporosis in perimenopausal
and postmenopausal women by using digital panoramic images,”
Oral Surgery, Oral Medicine, Oral Pathology, Oral Radiology, and
Endodontology, vol. 102, no. 2, pp. 235–241, 2006.
[7] P. Li, X. Yang, F. Xie, J. Yang, E. Cheng, V. Megalooikonomou, Y. Xu,
and H. Ling, “Trabecular texture analysis in dental cbct by multi-roi
multi-feature fusion,” in Biomedical Imaging (ISBI), 2014 IEEE 11th
International Symposium on. IEEE, 2014, pp. 846–859.
[8] H. Ling, X. Yang, P. Li, V. Megalooikonomou, Y. Xu, and J. Yang,
“Cross gender–age trabecular texture analysis in cone beam ct,”
Dentomaxillofacial Radiology, vol. 43, no. 4, pp. 20130324, 2014.
[9] C. Bo, X. Liang, P. Chu, J. Xu, D. Wang, J. Yang, V. Mega-
looikonomou, and H. Ling, “Osteoporosis prescreening using dental
panoramic radiographs feature analysis,” in Biomedical Imaging (ISBI
2017), 2017 IEEE 14th International Symposium on. IEEE, 2017, pp.
188–191.
[10] A. Krizhevsky, I. Sutskever, and G. E Hinton, “Imagenet classification
with deep convolutional neural networks,” in Advances in neural
information processing systems, 2012, pp. 1097–1105.
[11] G. J Burghouts and J. Geusebroek, “Material-specific adaptation of
color invariant features,” Pattern Recognition Letters, vol. 30, no. 3,
pp. 306–313, 2009.
[12] P. Mallikarjuna, and M. Fritz A. T. Targhi, E. Hayman, B. Caputo,
and J. Eklundh, “The kth-tips2 database,” 2006.
[13] S. Lazebnik, C. Schmid, and J. Ponce, “A sparse texture representation
using local affine regions,” IEEE Transactions on Pattern Analysis and
Machine Intelligence, vol. 27, no. 8, pp. 1265–1278, 2005.
[14] R. M Haralick, K. Shanmugam, et al., “Textural features for image
classification,” IEEE Transactions on systems, man, and cybernetics,
, no. 6, pp. 610–621, 1973.
[15] A. F. Costa, G. Humpire-Mamani, and A. J. M. Traina, “An efficient
algorithm for fractal analysis of textures,” in Graphics, Patterns
and Images (SIBGRAPI), 2012 25th SIBGRAPI Conference on. IEEE,
2012, pp. 39–46.
2582