SEGMENTATION OF FACES IN VIDEO FOOTAGE USING HSV COLOR
FOR FACE DETECTION AND IMAGE RETRIEVAL
Osamu Ikeda
Faculty of Engineering, Takushoku University
815-1 Tate, Hachioji, Tokyo, 193-0985 Japan
ABSTRACT
Previous studies on face detection in video footages show that
segmenting faces accurately and reliably i s often hard to
succeed, leading instead to laborious and tedious interactive
manipulation. This paper presents a segmentation method
using controlled weights on the three HSV components and
constructs aface detection and image retrieval system. First, it
is shown that HSV color has advantages over RGB or YCbCr
one when segmenting a face and generating a binary pattern
that retains as many features of the face as possible in the
original color picture. Then, a face detection and image
retrieval system i s constructed using HSV color, where each
time a significant scene change i s detected segmentation i s
carried out for the beginning frame using a few sets o f the
weights on the HSV components, and resulting patterns are
checked in some requirements and correlated with a typical
face pattern. Computer experiments show that the successful
detection rate i s more than 95 percent and that images can be
retrieved from an input face image in a short time.
1. INTRODUCTION
Research on face detection in images and its related areas has
extensively been made in recent years especially in the fields
of image processing and computer vision [I]. Rowley et al.
[2] and Feraud et al. [3] used neural network-based methods,
Schneiderman et al. developed a NaTve Bayes classifier [4],
Osuna et al. proposed an algorithm to train Support Vector
Machines [ 5 ] , and Turk et al. proposed to use eigenfaces [6].
Those algorithms are aimed at detecting or recognizing the
face in image. The detection is required to be in real time [7]
in computer vision, possibly at the sacrifice o f reliability for
each frame but not for a sequence o f frame images.
In the field o f multimedia, on the other hand, the focus has
been on not just its detection or recognition but also
identification o f faces, people, or some specific objects in
video images or video footages. Satoh e l al., for example,
tried to retrieve the name from the face or the face from the
name using the video, video caption and the transcripts [SI.
Since the segmentation accuracy affects to the identification
and the images may be available in a limited duration o f time,
several improvements have been reported. They combine
temporal segmentation or tracking with spatial segmentation
[9] or adopt manual segmentation [IO]. Long et al., for
0-7803-7750-8/03/$17.00 02003 IEEE Ill - 913
example, presented a method that uses three consecutive
frames to take into account motion and user interaction when
automatic detection fails [I I]. That may also be the case of
retrieving some visual information from video footages,
where accuracy may also be crucial.
In this paper, we present a segmentation method using HSV
color and construct a face detection and image retrieval
system. First, it i s shown that HSV color has advantages over
RGB or YCbCr in segmentation capability and easiness in its
manipulation. Then, a face detecting and image retrieving
system i s constructed based on the method, where each time a
significant scene change i s detected in the video footage,
segmentation is carried out for the beginning frame by
applying a few sets of the weights on the HSV components,
and resulting patterns are checked in some face requirements
and correlated with an input face pattem. Computer
experiments show that the successful detection rate is more
than 95 percent and that images in the video footage can be
retrieved from the input face image in a short time.
2. COLOR SEGMENTATION
We compare RGB, HSV and YCbCr colors experimentally to
choose the most reliable and convenient color for
segmentation. The segmentation means to extract the image
region continuous in the eight neighborhood with the sampled
point within a given error as:
where the suffix s denotes the sampled point, each color
component value i s normalized so as to vary between 0 and
unity, and IV'S are weights.
Face images in the video footages may have been
illuminated with a variety o f light sources, with a variety o f
color adjustments, and may include objects of similar colors
to that of the face in the background. In addition, they vary in
skin color, texture, frontal directions and size. So a single set
o f the weights on the color components may not be enough to
successfully segment faces for such a variety of images.
The desired segmented image may depend on applications;
whether such features as wrinkles are to be reflected in the
segmented image. In this paper, we require the segmented
image to include as many features as possible not just to
evaluate the segmentation capability but also to construct an
image retrieval system.
With this in mind, image regions which satisfy Eq. ( I ) but
are not continuous with the sampled point are included in the
segmented image, if even slightest parts of the regions are
longitudinally sandwiched within a certain distance by the
segmented part, as shown in Fig. 1. This extension may be
useful in making the segmented image more informative,
possibly, enough to recognize the person.
Fig. I The segmented region RA is
continuous with the sampled point
S, while the region RR in which
Eq. ( I ) is satisfied is not so. If
there exists a pixel in RB whose
distances D, and D2 to R, are
within a certain value, RH is
included in the seemented imaee.
3. FACE DETECTIONmETRIEVAL SYSTEM
The face detection or image retrieval is carried out based on
the segmentation method as follows:
(a) A significant scene change is detected in a video footage
(b) A sampling point is scanned on the beginning frame of a
new scene
(c) If the color at a sampled point is within a color window,
segmentation is carried out for a number of errors and for
a few sets of the weights on the HSV components
(d) The segmented image is made binary, which is then
checked in some requirements as face
(e) Then, the pattern is correlated with an input face pattern
(0 The segmented image with the largest correlation is
output as the face for the frame, where it also is possible
to detect multiple faces.
(g) Segmented face images are displayed according to their
correlation values.
The scene change is detected by evaluating the difference
between the neighboring frames:
Dxe, ( n )
where n is the order of the frame, i is the image intensity, and
&I 2 I , which is the transition number of frames from one
scene to another. The scene is judged to have changed if D,=
> Dlhholds.
The spacing of sampled points on the frame is determined
by the size of the face that we want to detect. Hoping to detect
faces whose sizes are larger than 8x12 pixels on a reduced
frame with (X,,Y,)=(76,58), we use 132 sampling points, 12
points in x direction and I I in y. At each point, first, the color
is checked whether it is in a relatively large (H,S,V I subspace
111 - 914
of the face image class for the case of face detection or in a
small subspace of a specific input image for the case of image
retrieval. Then, segmentation is carried out using a few sets of
the weights and six segmentation errors of 7, 9, 11, 13, 15,
and 17 percent. The weights are determined in advance based
on the results of manual segmentation, as will be shown.
The segmented image is required in size to be Y<2.5X and
X<I .25 Y. It is then made binary, and we impose the pattern to
satisfy the following conditions as face:
N , / X Y > ce for eyes
N , / X Y > cm for mouth and nose (3)
where N, and N,,, are numbers of non-segmented null pixels in
the regions designated as eye and mouth, respectively, in the
segmented image, as shown in Fig. 2, and c. and c,,, are
constants.
Fig. 2 Regions of eyes
and mouth for a
Y segmented image with
size X by Y.
X
Then, the binary pattern of the segmented image, P,,(x.y),
X,*~+X-I, y & ~ ~ + , + Y - is
l , correlated with an input face
pattern, P&,y), O~XSA'~-I,05~&Y~-l, as
a = (Xk-l)/(X -I), p = (Yk -l)/(Y -1)
where P., and Pk take I or -1 so that -l<C&l. The segmented
pattern is fitted in size to the input pattem. In the case of Y >
2X, where a neck part may often be included in the segmented
image, however, the fitting ratio in y is made the same as in x,
and the top part of the segmented pattern is used for the
correlation.
In the case of image retrieval, average R, G, B values are
obtained for each segmented image which are then correlated
with those ofthe input key face image as
where o is set to 0.1 to 0.2.
The segmented image with the largest value of C,, is
displayed for each new scene for the case of face detection,
while those with the largest values of C,,Cc are displayed for
the video footage for the case of image retrieval.
4. EXPERIMENTS
A video footage consisting of 100 frames shown in Fig. 3
was mostly used in the experiments, which may equivalently
consist of 100 different scenes. First, manual segmentation
was carried out using HSV, RGB and YChCr colors so as to
extract as many features of the faces as possible. Some of the
results are given in Fig. 4, which shows that HSV color can
most successfully segment the faces, while RGB or YCbCr
color fails to do so depending on the image. It also shows
that HSV color extracts more features than the others. The
control of the weights seems easier for HSV color than the
other two, since the HSV components have clear meanings.
Figure 5 shows four different cases of the optimal weights on
the HSV components; they are (a) iv>w,w,, (b) whw.>w,
(c) iv,=iv, (d) iv,>ivhiv, and ivh>iv,. That is, they v a v
depending not only on the face image but also on the
surroundings.
We used the three sets of the weights, W,=(l,I,l), W2=
(1,1,5) and W3=(2,2,1), to segment the 100 images in the face
detection and image retrieval system. Manually segmented
faces of the 100 images had mean values ranging from -20 to
61 in the scale of 360 for H, from 0.1 1 to 0.79 for S, and from
0.28 to 0.93 for V. So we set the color window to have the
color subspace [-20,611 for H, [O.I I , 0.791 for S, and 10.28,
0.931 for V for the case of face detection, and [h-8, h+8] for H,
[s-0.1,s+O.I] for S and [v-O.I,v+O.I] for V for the case of
image retrieval, where (h,s.v) are mean values of the input
face image.
The results are summarized in Table I for four input face
patterns, where the first and the second patterns are generated
from No.3 image in Fig. 3 with different segmentation errors
and have similar mean color values (12, 0.19, 0.69, the third
pattern generated from No.12 image has the means (39,0.79,
O S @ , and the fourth one generated from No. 45 image has the
means (16, 0.35, 0.77). It is seen from the results that more
than 95 faces out of 100 are successfully segmented, that the
rate tends to increase with increasing number of the weights,
and that using the multiple weights is more crucial for image
retrieval than for face detection.
The 100 segmented images for the second pattern and
weight (2,l,l) in Table I are shown in Fig. 6. It may be seen
that some segmented faces are cut off due to the restricted
segmentation region in size around the sampled point, and
that part of the background image is included in some of the
segmented images. The latter may be insignificant for face
detection but may affect significantly for image retrieval.
Figure 7 shows an example of detecting multiple faces, where
the two segmented regions are discriminated. The face
detection for the 100 face images took about 37 sec when
using a single set of the weights on a I.2MHz Pentiumlll PC,
and the image retrieval took one to eight sec depending on the
number of the weights used and the input image.
REFERENCES
[I] M-H. Yang, D.J. Kriegman, N. Ahuja, “Detecting Faces in
Images: A Survey,” IEEE Trans.PAMl,vol.24.pp.34-58,2002.
[2] H.A. Rowley, S. Raluja, T. Kanade, “Neural Network-Based
Face Detection,” IEEE Trans.PAMI, ~01.20, pp.23-38, 1998.
[3] R. Feraud, O.J. Bemier, I.-E. Viallet, and M. Collobert, “A
Fast and Accurate Face Detector Based on Neural Networks,”
IEEETrans. PAMI, vol. 23, pp. 42-53, 2001.
[4] H. Schneiderman and T. Kanade, “Probabilistic Modeling of
Local Appearance and Spatial Relationships for Object
Recognition,.’ Proc. CVPR, pp. 45-51, 1998.
[5] E. Osuna R. Freund, and G Girosi, “Training Support Vector
Machines: An Application to Face Detection,” Proc. CVPR,
pp. 130-136, 1997.
[6] M.A. Turk and A.P. Pentland, “Eigenfaces for pattern
recognition,” J.Cognitive Neuroscience,vol.3.pp.71-96, 1991.
[7] B. Fr(lba, A. Emst, and C. Kilblbeck, “Real-Time Face
Detection,” Proc. 4Ih IASTED International Conf. Signal and
Image Processing, pp. 497-502, 2002.
[8] S. Satoh, Y. Nakamura and T. Kanade, “Name-It: Naming
and Detecting Faces in News Videos,” IEEE Multimedia, pp.
22-35, 1999.
191 D. Wang, “Unsupervised Video Segmentation Based on
Watersheds and Temporal Tracking,” IEEE Trans. Circuits
and Systems for Video Technology, vo1.8, pp.539-545, 1998.
[IO] C. Toklu et al.. “Simultaneous Alpha Map Generation and
2D Mesh Tracking for Multimedia Applications,“ Proc.
KIP, vol. 1. pp. 113-116,1997
[ I l l E Long, D. Feng, H. Peng, and W. Siu, “Extracting
Semantic Video Objects,” IEEE Computer Graphics and
Applications, pp. 48-55.2001.

5. CONCLUSIONS
The segmentation method using HSV color was shown to be
more accurate and easier for face images compared with RGB
or YCbCr color. Based on this method, we constructed a face
detectioniimage retrieval system. Using a few sets of the
weights on the three HSV components it was able to detect
more than 95 faces out of 100 successfully, and it could
retrieve images from the input face image in a short time.

Fig. 3 A video footage consisting of 100 frames

111 - 915
 Table 1. Detection rates for the 100 face images in Fig. 3 for four
input patterns, and identification results for the same data where
the color subspaces are different between the two experiments.

Fig. 4 Comparison of segmented images obtained using HSV,

RGB and YCbCr colors for four face images. The weights on the
three components are given in the parenthesis and the
segmentation error is given in percent.

Fig. 6 Segmented images for the 100 face images. where the
pattern for tlSV in Fig. 4(a) was used as input face pattern.

(C) (d) Fig. 7 Example ofmulti-face detection on the 42nd frame in Fig.
Fig. 5 Four cases of optimal weights on HSV: from top-left to 3.
bottom-right, (wh, w., w,,)=(l,lO,l), (lO,l,lO), (l,l,lO),(l,l,l),
(lO.IO.l), ( l O , l , l ) , ( l , I O . l O ) for (a), (b) and (d) and ( l , 3 , l ) ,
(l0,3,l0),(l,l.3),( l , l , l ~ , ~ l O , l O , 3 ) , ( l O , 3 , 3 )(1.3.3)for(c).
,

111 - 916