2018 the 3rd IEEE International Conference on Cloud Computing and Big Data Analysis

Real-Time Object Recognition Algorithm Based on Deep Convolutional Neural

Network

Lihong Yang, Liewei Wang, Shuo Wu

The Thirty-eighth Research Institute of China Electronic Technology Group Corporation
Hefei Research Institute of Public Safety Technology
Hefei, China
e-mail: lih.yang@163.com, levidwang@163.com, perfectshuo@126.com

Abstract—Radar detection of moving objects is vulnerable to
external environment. By introducing the object confirmation
algorithm, the intelligent radar perimeter security system’s
false alarm rate can be reduced significantly. The object
confirmation algorithm is essentially an object detection
algorithm. Due to the poor generalization of artificial feature
extraction algorithms, we use the deep convolution neural
network to extract deep features automatically for object
confirmation. In order to meet the real-time requirement in
engineering practices, our algorithm use the YOLOv2 system
as a basis, and selects anchor boxes which meet object scales of
our training data set by k-means++ clustering. To improve the
YOLOv2 network structure, low-layer deep features which
denote the texture information and high-layer deep features
which denote the semantic information are combined layer by
layer to make object detection more accurate. The
experimental results show that the false alarm rate of the
intelligent radar perimeter security system is further reduced
by introducing the object confirmation algorithm. Especially
for extreme weather, false alarms of radar detection greatly
increase. But most of them are eliminated after running object
confirmation algorithm. Therefore the warning accuracy of the
entire system can be guaranteed. The detection speed of the
object confirmation algorithm is 33FPS, which meets the real-
time requirement of engineering practices.
Keywords-YOLOv2; deep convolutional neural network;
feature fusion; perimeter security
I. INTRODUCTION
Perimeter security system is widely used in airports,
nuclear power plants, oil fields, prisons, etc., to prevent
illegal invasion. The intelligent radar perimeter security
system is a next-generation product which is developed to
reduce the high false alarm rate of the traditional perimeter
security system. Due to external interference signals caused
by swaying trees and moving animals, radar detection still
contains some false alarms. By introducing the object
confirmation algorithm, false alarms detected by the radar
can be eliminated and the warning accuracy can be further
improved.
Object confirmation essentially is object detection based
on an image. Traditional object detection algorithms include
three steps: region selection, feature extraction and
classification. Feature extraction is the key factor which
affects the system performance. Because of the poor
generalization of widely used features such as HOG and
SIFT, features applied to engineering practices should be
designed specifically. Therefore, the detection accuracy
depends heavily on the experience of developers. In order to
obtain higher accuracy, multiple features are used, so the
feature dimension becomes bigger and bigger which greatly
reduces the real-time performance of algorithms. As a result,
it is difficult to make a breakthrough in practice for
traditional object detection algorithms.
With the continuous construction of large-scale data sets
and the ever-increasing hardware computing power, theories
and practices of deep learning have been rapidly developed
in recent years. Object detection algorithms based on deep
convolutional neural networks have achieved a qualitative
improvement in performance. In 2014, deep features are
applied to object detection for the first time in R-CNN [1].
Since then, object detection with deep learning based on
Region Proposal was pioneered. SPP-net [2], Fast R-CNN
[3], Faster R-CNN [4] and R-FCN [5] all were improvement
of this kind of method. With the optimization of deep
convolutional neural network, the above methods achieved
higher accuracy and faster detection speed. But all of them
didn’t achieve real-time. In engineering practices, real-time
is very important. As a result, object detection algorithms
with deep learning based on regression were proposed.
YOLO [6], SSD [7] and YOLOv2 [8] were typical
representatives. Without extraction of region proposals, such
methods directly obtained the location and category of
objects by regression, achieving real-time detection.
Compared with traditional feature extraction methods,
the deep convolution neural network simulates the human
brain. By continuous learning, deep features are extracted
from images automatically, which not only avoid
complicated feature design step but also have powerful
generalization. According to the real-time requirement of the
object confirmation module, our object confirmation
algorithm is based on YOLOv2. Our training data set is
constructed and anchor boxes are selected by clustering for
this training set. For network structure, the detail texture
information of deep features in low layers and the semantic
information of deep features in high layers are combined to
make object detection more accurate. The experimental
results show that most of false alarms captured by the radar
are eliminated by introducing our object confirmation
algorithm. The false alarm rate of the system is reduced

978-1-5386-4301-3/18/$31.00 ©2018 IEEE 331

while the accuracy of the system is improved. The average
detection speed is 33FPS, which satisfies the real-time
requirement of engineering practices.
II. INTRODUCTION OF YOLOV2
YOLOv2 reframes object detection as a single regression
problem, straight from image pixels to bounding box
coordinates and class probabilities simultaneously by
running a single deep convolutional neural network on the
image. Because YOLOv2 unifies bounding box extraction,
feature extraction, object classification and object location
into a single network, it not only enables end to end
optimization during training, but also achieves real-time
when detecting.
When predicting the location and class of each object, the
input image is zoomed to a uniform size and divided into M
× M cells. For each cell, with given B anchor boxes, B
bounding boxes are predicted after running a single deep
convolution neural network. For each bounding box, the
network predicts five coordinates and C probabilities. By
non-maximum suppression, the final objects are obtained
among the M × M × B bounding boxes.
The deep convolutional neural network used by YOLOv2
is a fully convolutional network with 19 convolutional layers
and 5 pooling layers. Inspired by the global average pooling
in Network in Network (MIN) [9], 1x1 reduction layer are
used between two 3x3 convolutional layers to compress deep
features. When applied to VOC (The PASCAL Visual
Object Classes) data set, M = 13, B = 5, C = 20, so the final
output of the network is 13 × 13 × (5 × (5 + 20)) = 13 × 13 ×
125. The full network is shown in Fig. 1.

Figure 1. The deep convolutional neural network architecture of YOLOv2.

III. OBJECT CONFIRMATION ALGORITHM

We divide the objects needed to detect in our project into
six classes, named people, car, motorbike, minibus, truck and
bus. The entire algorithm is divided into the training step and
detecting step.
During the training step, we collect a large number of
images with objects belong to the above six classes and label
them. Then, we design a deep convolution neural network
and initialize the network with parameters which is obtained
by pre-training on the large-scale data set. The above images
are fed into the deep convolutional neural network to predict
locations and classes of objects in them. By back-
propagation algorithm, the difference between predicted Figure 2. The diagram of the detecting step.
results and ground truths is continuously reduced. Finally,
we obtain the deep model for object confirmation as a result
of the training step, which is used to initialize the deep
convolution neural network in the detecting step.
The diagram of the detecting step is shown in Fig. 4.
Firstly, the image is zoomed to 416 × 416 and divided into
13 × 13 detection cells. Then 9 bounding boxes are predicted
for each detection cell, so a total of 1521 bounding boxes are
obtained for the whole image. Finally, by non-maximum
suppression, objects in the image which belong to the six
classes are obtained like the black car and the pedestrian
shown in Fig. 4.
Figure 3. Average IOU plot with different number of clustering centers.

332

Figure 4. Feature Combined module.
IV. TRAINING DATA SET
A. Training Data Set Construction
The training data set is a very important factor whether
deep learning can get good results. Therefore, in the training
step, the training data set construction is a key task. In our
project, we collect a large number of images as training
samples, which contain objects of the six classes named
people, car, motorbike, minibus, truck and bus. The location
and class of each object in the images are labeled. To ensure
the detection accuracy, training samples should be as diverse
as possible. The images in our training data set not only
include different outdoor scenes such as security monitoring
and traffic, but also cover different light environments such
as sunny day, rain, snow, and night. In addition, scales and
deformations of objects also should be as diverse as possible.
B. Anchor Boxes Slection
It is easier to achieve stable training of the deep model by
predicting the offset of each bounding box to related
detection cell than directly calculating coordinates of each
bounding box. So it is necessary to provide anchor boxes
with specified scales as the reference to predict bounding
boxes.
YOLOv2 provides anchor boxes for training sets like
VOC and COCO (Common Objects in Context). Because
there are great differences in object classes and scenarios
between our training data set and them, we must calculate
anchor boxes which are suitable for our training data set.
We collect the width and height data of all objects in our
training data set. Then we run k-means++ clustering on them
to get anchor boxes which can best fit the aspect ratio of our
training data set. The detection accuracy is measured by the
IOU (Intersection over Union) between the bounding box
and the ground truth, so we use the distance metric shown in
The number of anchor boxes is equal to the number of
clustering centers. For various numbers of clustering centers,
the average IOU between each ground truth and its closet
centroid is shown Fig. 5. When using 9 clustering centers,
the upward trend of the average IOU has tended to be stable.
Therefore, we use 9 different anchor boxes to balance the
detection accuracy and the model complexity. Table I shows
the width scale and height scale of each anchor box which
refer to the ratio of the width and height of each anchor box
to the width and height of related detection cell whose width
and height are both 32pixels.
V. NETWORK STRUCTURE
Deep convolution neural network is the soul of deep
learning. A well-designed network can efficiently extract
deep features of objects. Because the low-layer filters extract
the detail texture information of objects while the high-layer
filters extract the semantic information, multi-feature fusion
has become a new trend in the deep convolutional neural
network design in recent years [10]. Combining the semantic
information with the texture information for object detection
can achieved higher detection accuracy.
TABLE I. ANCHOR BOXES WITH 9 CLUSTERING CENTERS
Anchor box Width scale Height scale
1 0.63 1.22
2 0.92 3.12
3 1.68 2.00
4 1.73 5.21
5 2.96 4.32
6 3.00 8.59
7 4.87 6.80
8 5.77 10.38
9 10.36 10.70
A. Feature Combined Module
YOLOv2 combines the 26 × 26 × 256 features before the
last pooling layer with the subsequent 13 × 13 × 1024
features to obtain 13 × 13 × 3072 high-dimensional features
for the final detection. In order to excavate more deep
features, the feature combined module shown in Fig.4 is used
to merge the low-layer deep features one by one with the
subsequent high-layer deep features. The fusion process is
carried out along the deep convolutional neural network
layer by layer.
As shown in Fig. 4, Firstly we perform batch
normalization on the previous depth feature named
feature_N and then put the output to conv1. Secondly, we
apply the second batch normalization to the output of conv1
and then put the output to conv2. Finally, we combine the
output of conv2 with feature_N to obtain the combined
feature named feature_N + 1.
d(box, centroid) = 1 IOU(box, centroid)

333
B. Network Structure
The structure of our deep convolutional neural network is
shown in Fig. 5. Before each pooling layer, the low-layer
features are superposed on the high-layer deep features by
multiple feature combined modules and then fed to 1 × 1
convolution layer which has gradually reduplicated the
number of filters.
We should detect objects with six different classes. The
input image is divided into 13 × 13 detection cells and 9
bounding boxes are predicted for each cell. Therefore, our
final output is 13 × 13 × (9 × (5 + 6)) = 13 × 13 × 99.

Figure 5. Our deep convolutional neural network structure.

VI. DEEP CONVOLUTION NEURAL NETWORK

The running environment of our object confirmation
algorithm is as follows: CPU is Inter Xeon CPU E5-2623 v3
3.00GHz with 64G memory; GPU is NVIDIA Geforce
GTX1080Ti; the operating system is 64-bit Windows 10
Enterprise Edition.
Under the above condition, average detection time per
frame is 30ms, which fully meets the real-time requirement
of the intelligent radar perimeter security system. (b) Shaking trees under the snow
Radar object detection is vulnerable to the natural
environment. Fig. 6 shows the false alarms detected by the
radar. 6(a) is caused by shaky leaves in the wind; 6(b) is
caused by the snow on the tree; 6(c) is caused by the drift of
rain and snow in the extreme weather. When the above
images are input to the object confirmation module, because
there aren’t interested objects such as persons or vehicles in
them, the false alarms are removed from warning results.
Therefore the system’s false alarm rate is reduced.
Table II shows warning results of the intelligent radar
(c) Drift of rain and snow
perimeter security system for test on January 5, 2018,
January 6, 2018 and January 7, 2018. Figure 6. False alarms detected by the radar.

TABLE II. OBJECT CONFIRMATION RESULTS

Object confirmation
Date Radar warnings
warnings
20180105 2860 2722

20180106 1177 903

20180107 1768 1103

(a) Shaking leaves in the wind

334
 Because the weather condition on January 5, 2018 was
stable, there were 148 false alarms in 2860 warnings. After
object confirmation, 138 false alarms were removed, that is,
93.3% of false alarms were eliminated. Because of the
frequent rocking of trees due to snow melting on January 6,
2018, false alarms increased significantly. There were 304
false alarms in 1177 warnings. After object confirmation,
274 false alarms were removed, that is, 90.1% of false
alarms were eliminated. There were 719 false alarms in 1768
warnings on January 7, 2018. After object confirmation, 651
false alarms were removed, that is, 90.5% of false alarms
were eliminated. In conclusion, under stable weather
conditions, the intelligent radar perimeter security system
can achieve effective warning even in complex scenarios. Its
false alarm rate can be further reduced by introducing the
object confirmation module. Under extreme weather
conditions, false alarms detected by the radar will be
increased substantially. At this time, with the object
confirmation module, most of false alarms can be eliminated
and the entire system’s warning accuracy can be guaranteed.
VII. CONCLUSION
By introducing the real-time object confirmation
algorithm based on deep convolution neural network, the
intelligent radar perimeter security system incorporates the
high sensitivity of radar detection and the high accuracy of
object confirmation. Due to breaking the limitation of single
technology, the system’s false alarm rate is reduced while the
system’s accuracy is improved. Using the object
confirmation algorithm, the intelligent radar perimeter
security system significantly reduces the dependence on the
weather environment. It can still achieve effective warning
under adverse weather conditions.
ACKNOWLEDGMENT
This work is financially supported by National Key
Research and Development Project of China through grant
335
2017YFC0804900, the National Natural Science Foundation
of China through grant 61503352.
REFERENCES
[1] R. Girshick, J. Donahue, T. Darrell and J. Malik, “Rich Feature
Hierarchies for Accurate Object Detection and Semantic
Segmentation,” Proc. IEEE Conf. Computer Vision and Pattern
Recognition (CVPR 14), IEEE Press, Jun. 2014, pp. 580-587, doi:
10.1109/CVPR.2014.81.
[2] K. He, X. Zhang, S Ren and J. Sun, “Spatial Pyramid Pooling in Deep
Convolutional Networks for Visual Recognition,” IEEE Transactions
on Pattern Analysis and Machine Intelligence, vol. 37, Sept. 2015, pp.
1904-1916, doi: 10.1109/TPAMI.2015.2389824.
[3] R. Girshick, “Fast R-CNN,” Proc. IEEE Conf. Computer Vision
(ICCV 15), IEEE Press, Dec. 2015, pp. 1440-1448, doi:
10.1109/ICCV.2015.169.
[4] S Ren, K. He, R. Girshick and J. Sun, “Faster R-CNN: Towards Real-
Time Object Detection with Region Proposal Networks,” IEEE
Transactions on Pattern Analysis and Machine Intelligence, vol. 39,
Jun. 2017, pp. 1137-1149, doi: 10.1109/TPAMI.2016.2577031.
[5] J. Dai, Y. Li, K. He and J. Sun, “R-FCN: Object Detection via
Region-based Fully Convolutional Networks,” arXiv:1605.06409, Jun.
2016, unpublished.
[6] J. Redmon, S. Divvala, R. Girshick and A. Farhadi, “You Only Look
Once: Unified, Real-Time Object Detection,” Proc. IEEE Conf.
Computer Vision and Pattern Recognition (CVPR 16), IEEE Press,
Jun. 2016, pp. 779-788, doi: 10.1109/CVPR.2016.91.
[7] W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed et al., “SSD:
Single Shot MultiBox Detector,” Proc. 14th European Conference.
European Conference on Computer Vision (ECCV 2016), Springer
Press, Sept. 2016, pp. 21-37, doi: 10.1007/ 978-3-319-46448-0_2.
[8] J. Redmon and A. Farhadi, “YOLO9000: Better, Faster, Stronger,”
Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR
17), IEEE Press, Jul. 2017, pp. 6517-6525, doi:
10.1109/CVPR.2017.690.
[9] M. Lin, Q. Chen and S. Yan, “Network In Network,” Proc.
International Conference on Learning Representations(ILCR 2014),
Apr. 2014.
[10] K. He, X. Zhang, S Ren and J. Sun, “Deep Residual Learning for
Image Recognition,” Proc. IEEE Conf. Computer Vision and Pattern
Recognition (CVPR 16), IEEE Press, Jun. 2016, pp. 770-.787, doi:
10.1109/CVPR.2016.90.