I. Yoon et al.

: Adaptive Defogging with Color Correction in the HSV Color Space for Consumer Surveillance System 111
Manuscript received 01/15/12
Current version published 03/21/12
Electronic version published 03/21/12. 0098 3063/12/$20.00 2012 IEEE
Adaptive Defogging with Color Correction in the HSV Color Space
for Consumer Surveillance System
Inhye Yoon, Seonyung Kim, Donggyun Kim, Member, IEEE, Monson H. Hayes, Fellow, IEEE,
and Joonki Paik, Member, IEEE

Abstract Consumer video surveillance systems often
suffer from bad weather conditions, observed objects lose
visibility and contrast due to the presence of atmospheric
haze, fog, and smoke. In this paper, we present an image
defogging algorithm with color correction in the HSV color
space for video processing. We first generate a modified
transmission map of the image segmentation using multiphase
level set formulation from the intensity (V) values. We also
estimate atmospheric light in the intensity (V) values. The
proposed method can significantly enhance the visibility of
foggy video frames using the estimated atmospheric light and
the modified transmission map. Another contribution of the
proposed work is the compensation of color distortion
between consecutive frames using the temporal difference
ratio of HSV color channels. Experimental results show that
the proposed method can be applied to consumer video
surveillance systems for removing atmospheric artifacts
without color distortion
1
.

Index Terms Single image defogging, color correction,
enhancement of the video surveillance system, bad weather.
I. INTRODUCTION
Identification of the original color and shape of an object is
a challenging problem in consumer video surveillance
systems, and the problem becomes even worse with unclear
meteorological conditions including fog, smoke, storm, and
snow. For overcoming this problem, various image processing
approaches have been investigated.
For example, Narasimhan et al. [1] addressed the problem
of restoring the contrast of atmospherically degraded images
and videos. Narasimhans method estimates depth from two
images of the same scene that are captured under different

1
This research was supported by the Chung-Ang University excellent
freshman scholarship grants, by Business for Cooperative R&D between
Industry, Academy, and Research Institute funded by Korea Small and
Medium Business Administration in 2011 (00045420-1), and by Basic
Science Research Program through National Research Foundation (NRF) of
Korea funded by the Ministry of Education, Science and Technology (2009-
0081059).
Inhye Yoon is with the Department of Image, Chung-Ang University,
Seoul, Korea (e-mail: inhyey@gmail.com).
Seonyung Kim is with the Department of Image, Chung-Ang University,
Seoul, Korea (e-mail: sy613h@gmail.com).
Donggyun Kim is with the Department of Image, Chung-Ang University,
Seoul, Korea (e-mail: deepain83@gmail.com).
Monson H. Hayes is with the Department of Image, Chung-Ang
University, Seoul, Korea (e-mail: mhh3@gatech.edu).
Joonki Paik is with the Department of Image, Chung-Ang University,
Seoul, Korea (e-mail: paikj@cau.ac.kr).
weather conditions. In spite of the improved defogging
performance, this method cannot be used for dynamic scenes
because of the requirement to capture multiple images of the
same scene under different environmental conditions.
Shwartz [2] exploited two or more images of the same scene
having different degrees of polarization by rotating a
polarizing filter attached to the camera. This method is very
constrained in the image acquisition process, and cannot be
used on existing image databases. Hautiere [3] used multiple
images for the same scene under different weather
conditions to compute depth information for contrast
enhancement. Although existing methods addressed the
possibility of enhancing foggy images, they are not suitable
for consumer cameras because of the need of multiple
images and special hardware devices.
From a theoretical point of view, defogging can be
considered as an under-constrained problem if only a single
foggy image is available. Fattal [4] estimated the albedo of
the scene and then inferred the medium of the transmission
under the assumption that the transmission and surface
shading are locally uncorrelated. However, this approach
cannot handle images that are heavily foggy, and may fail if
the uncorrelated assumption does not hold. Tan [5] observed
that the fog-free image must have higher contrast than the
unprocessed version of the foggy image. Based on this
observation the foggy component is removed by maximizing
the local contrast of the restored image. Chen [6][7]
observed that most local regions of a fog-free image have a
set of pixels with very low intensity, and used the dark
channel prior to remove fog. Kratz [8] used a factorial
Markov random field to model the haze image, and takes
scene albedo and scene depth as two statistically
independent components, and removed the haze by
factorizing the image into scene albedo and depth. However,
the single image-based methods are prone to halo effect and
color distortion problem.
Fog removal method is similar to the contrast enhancement
method. Kong [9] conducted histogram equalization over all
image pixels concurrently. On the other hand, local
equalization tackles image enhancement by dividing the image
into multiple sectors and equalizing them independently. Xu
[10] used the parameter-controlled virtual histogram
distribution method, and enhanced both overall contrast and
sharpness of an image. While it can increases the visibility of
specified portions or aspects of the image color, this approach
cannot correctly restore the original color in the foggy image.
112 IEEE Transactions on Consumer Electronics, Vol. 58, No. 1, February 2012
In order to solve the aforementioned problems, we present a
novel video defogging algorithm with color correction for the
consumer video surveillance system. The proposed algorithm
first generates a modified transmission map using the
multiphase level set formulation from the intensity (V) values
in the HSV color space, and also estimates the atmospheric
light from the intensity (V) values. Based on the estimated
atmospheric light and the modified transmission map, we can
simultaneously remove fog and correct color distortion. The
proposed color correction method compensates color
distortion between consecutive frames using the temporal
difference ratio of HSV color channels. Experimental results
demonstrate that the proposed algorithm not only increases the
visual quality of atmospherically degraded video inputs, but
also improves the performance of object detecting and
tracking in consumer video surveillance systems.

II. FOGGY IMAGE FORMATION MODEL
A scene is produced if one or more objects are illuminated
by the light source as shown in Fig. 1. The light reflected from
an object is scattered and absorbed by aerosols in the
atmosphere before it reaches the observer. More specifically,
the light arriving at the camera consists of two components; i)
directly attenuated light and ii) airlight. The former represents
the original scene components partially attenuated by
absorption in the atmosphere, and the latter the light reflected
from other directions by scattering in the atmosphere. Because
of the atmospheric absorption, the original color of an object
is changed into a foggy color. The quality of consecutive
frames in bad weather is usually degraded by the presence of
fog in the atmosphere, since attenuation in the incident light
decreases the contrast of the acquired image. Fig. 1 shows the
foggy image formation model.


Fig. 1. The optical model of foggy image acquisition.

In the computer vision research field, the image degradation
model due to fog is described in RGB color space as

( , ) ( , ) ( , ) (1 ( , ))
c c
I O
f x y f x y T x y A T x y ,
for { , , } c R G B , (1)
where ( , )
c
I
f x y represents the foggy image, ( , )
c
O
f x y

the
ideal or original fog-free image, A

the global atmospheric
light, and ( , ) T x y

the transmission map that is independent of
the color.
In the right hand side of (1), ( , ) ( , )
c
O
f x y T x y

represents the
directly attenuated component, and the second term
(1 ( , )) A T x y

represents the airlight. More specifically, the
direct attenuated component describes the decayed version of
the ideal fog-free image ( , )
c
O
f x y , while airlight results from
scattered light in other directions and color shifts in the scene.
The defogging problem is to restore ( , )
c
O
f x y , given
( , )
c
I
f x y . In the restoration process, it is necessary to
estimate A

and ( , ) T x y .

III. THE PROPOSED ALGORITHM
The proposed video defogging algorithm consists of
transmission map generation, atmospheric light estimation,
recovery of defogged frame, and color correction using
consecutive frames, as shown in Fig. 2, where time variable t

is added for extending the image formation model in (1) to the
video.


Fig. 2. The proposed video defogging algorithm for consumer video
surveillance systems.

Each step of the algorithm is described in the following
subsections. In this section, we use a set of test images; an
ideal fog-free image and its foggy version to demonstrate the
performance of the proposed algorithm as shown in Fig. 3.


(a) (b) (c)
Fig. 3. Test images; (a) original fog-free image, (b) the simulated foggy
image, and (c) the defogged image using the proposed method.

A. Modified transmission map generation using multiphase
level set
Existing methods for generating the transmission map
search for the lowest intensity in the patch centered at ( , ) x y ,
I. Yoon et al.: Adaptive Defogging with Color Correction in the HSV Color Space for Consumer Surveillance System 113
called ( , ) x y O , from (1) as [6]

( )
( , ) ( , )
( , )
{
( , )
, , }
{ , , }
( , )
min min
( , )
( , ) min min 1 ( , ) .
c
I
R G B
c
O
c
R G
p q x y
c p x y B q
f p q
A
f p q
T x y T x y
A
eO
e
e
O e
| |

` |
\ . )
| |

= +
` |
\ . )
(2)

Ideally, the minimum intensity in a patch of the ideal fog-
free image should be zero. Therefore, we compute the
transmission map as


{ , , } ( , ) ( , )
( , )
( , ) 1 min min
c
I
R c p y B q G x
f p q
T x y
A
eO e
| |

=
` |
\ . )
. (3)

We note that existing transmission map exhibit a halo effect
and color distortion since the intensity discontinuity across
edges is not considered in the reconstruction process.
Existing halo effect reducing method uses image matting,
but it causes extremely high computational complexity, which
is unsuitable for on-line video processing. To solve this
problem, we generate a modified transmission map using the
multiphase level set formulation, which partitions the image
into piecewise uniform regions. Since the transmission map
determines how much the light reflected by the object reaches
the camera, we assume that the light traveling a longer
distance is more attenuated, which yield the modified
transmission map as


( , )
( , )
d x y
x y e T
|
=

, (4)

where |

is the scattering coefficient of the atmosphere, and
( , ) d x y

represents the depth or distance of the location ( , ) x y .
For obtaining the depth map in (4), the multiphase level set
method is used to reduce the processing time.
Let multiple level set functions { , 1,......, }
j
j n | =

represent
the rgions { , 1,......, }
j
j N O =

with 2
n
N =

as defined in [11].
The j -th level set function
j
|

is typically defined as the
signed distance function of the corresponding region
j
O . We
compute the depth map ( , ) d x y using two level set functions
that generates four phases as shown in Fig. 4.


Fig. 4. Partitioning of the image into four phase using two curves.
The four phases provide an effective, implicit
representation for evolving curves and surface. Therefore, we
compute the four phases as


1 1 2 2 1 2
3 1 2 4 1 2
( ) ( ), ( )(1 ( ))
(1 ( )) ( ), (1 ( ))(1 ( ))
P H H P H H
P H H P H H
| | | |
| | | |
= =

= =

, (5)

where ( )
j
H |

is the Heaviside function. The depth map
( , ) d x y

in each variable is then obtained by fixing other
variables as

4
4
( )
( , )
V j j
j
j j
j
f P c
d x y
P c

, (6)

where
v
f

represents the intensity (V) values in the HSV color
space. The modified transmission map can be generated from
the depth map obtained in (6). The conventional and modified
transmission maps of Fig. 3(b) are shown in Fig. 5(a) and Fig.
5(b), respectively. Since the intensity values are discontinuous
at the boundary of each pixel, we used a multiphase level set
to mitigate the discontinuities. As a result, the proposed
modified transmission map has continuous intensity values in
the neighborhood of boundaries.


(a) (b)
Fig. 5. Two different transmission map generated by; (a) the existing
method [4] and (b) the proposed method.

B. Estimation of atmospheric light and image defogging
Existing atmospheric light estimation methods extract the
brightest pixel among all color channels in the foggy image,
which results in the color distortion problem if there are
originally white objects without atmospheric degradation. In
this paper, we use the highest intensity (V) value in the HSV
color space for the atmospheric light [5]. Given the
atmospheric light A

and the modified transmission map
( , ) T x y

, the V component of the defogged image frame can

be recovered as

( , , ) ( )
( , , ) ( )
( , , )
V
V I
D
f x y t A t
f x y t A t
x y t T

= +

(7)

where t

represents the frame number.
114 IEEE Transactions on Consumer Electronics, Vol. 58, No. 1, February 2012
Since it is not easy to quantify the performance of
defogging, we provide the defogged image the proposed
method in Fig. 3(c). Extensive comparison with existing
methods will be presented in section 4.

C. Color Correction between Consecutive Frames
After removing the foggy components, each frame may
have different color tones because the atmospheric light
and the transmission map are generated without
consideration of the temporal correlation between
consecutive frames. For solving this problem, it is
necessary to perform color correction for the defogged
frames to reduce the changes in color tone between
frames.
In the HSV color space, given a certain intensity (V)
value saturation (S) is defined as the relative length of the
vector that points to the given color on the border, And
hue (H) is defined as the linear length along the loci of
constant Saturation (S) from the reference vector
indicating the red color, that is the vector r ,

as shown in
Fig. 6 [12].



(a) (b) (c)
Fig. 6. Saturation and hue of a color vector as defined in the HSV color
space; (a) intensity values, (b) saturation color channel, and (c) hue color
channel.

For the proposed color correction use compute frame
differences in all HSV color channels as


{ }
1
( , , ) ( , , 1)
M N
d
x y
d d
D D
c f x y t f x y t
M N
=

,
for { , , } H d S V e , (8)

where M and N represents horizontal and vertical sizes of
the image, respectively. We can then obtain color corrected
HSV color channels as

( , , ) ( , , )
d d d
E D
f x y t f x y t c = + , for { , , } H d S V e . (9)

By transforming the processed HSV color space, which are
H
E
f ,
S
E
f , and
V
E
f

in (9), into the RGB color space, we finally
obtain the enhanced video frames. Fig. 7(a) shows foggy input
video frames captured by a surveillance camera. The defogged
and color corrected frames are shown in Fig. 7(b) and Fig.
7(c), respectively.


(a)

(b)

(c)
Fig. 7. Experimental results of defogging and color correction method for
video surveillance system; (a) the input consecutive frames (40th, 60th,
80th, and 100th frames), (b) the defogged frame by using the proposed
method, and (c) the defogged and color corrected frames by using the
proposed method.

IV. EXPERIMENTAL RESULTS
In this section, we demonstrate the performance of the
proposed algorithm for enhancing foggy video.
Fig. 8(a) shows another input foggy video frames. The
results of the modified transmission map using the proposed
method, the defogged frames using the proposed method, and
the color corrected frames using the proposed method are
shown in Fig. 8(b), Fig. 8(c), and Fig. 8(d), respectively.
Experimental results demonstrated that the proposed
algorithm outperforms the existing algorithm in the sense of
both preserving the original color and improving visibility.


(a)

(b)

(c)

(d)
Fig. 8. Experimental results of defogging with color correction method;
(a) the input consecutive frames (90th, 100th, 110th, and 120th
frames),(b) the modified transmission map by using the proposed method,
(c) the defogged frames by using the proposed method, and (d) the
defogged and color corrected frames by using the proposed method.
I. Yoon et al.: Adaptive Defogging with Color Correction in the HSV Color Space for Consumer Surveillance System 115
Fig. 9(a) shows a set of foggy video frames, and Fig. 9(b)
shows the defogged frames using the existing method
proposed in [4]. The results of modified transmission maps
using the proposed method are shown in Fig. 9(c) and Fig.
9(d) shows the defogged frames using the proposed algorithm.
Based on the experimental results, the proposed defogging
method significantly outperforms existing method in the sense
of both contrast recovery and color preservation.


(a)

(b)

(c)

(d)
Fig. 9. Experimental results of various defogging methods; (a) an input
foggy image, (b) the defogged image by using the existing method in [4],
(c) the modified transmission map by using the proposed method, and (d)
the defogged image by using the proposed method.

V. CONCLUSIONS
In this paper, we proposed a combined video defogging and
color correction methods in the HSV color space.
The proposed algorithm first generates a modified transmission
map using the multiphase level set formulation in the intensity
(V) component of the HSV color space, and estimates the
atmospheric light the intensity (V) values. Defogged frames
are then produced using the atmospheric light and th
e modified transmission map. Finally, color distortion between
consecutive frames is corrected by using the tem
poral difference ratio of HSV color channels. Experimental results
demonstrated that the proposed algorithm outperforms the
existing algorithm in the sense of both preserving the original
color and improving the visibility. The proposed method can
be applied to the extended areas of image enhancement by
removing fog-like haze, clouds, and dust from the consumer
video surveillance systems.
REFERENCES
[1] S. Narasimhan and S. Nayar, Contrast restoration of weather
degraded images, IEEE Trans. Pattern Analysis and Machine
Intelligence, vol. 25, no. 6, pp. 713-724, June 2003.
[2] S. Shwartz, E. namer, and Y. Schechner, Blind haze separation,
Proc. IEEE Int. Conf. Computer Vision, Pattern Recognition, pp.
1984-1991, October 2006.
[3] N. Hautiere, J. Tarel, and D. Aubert, Towards fog-free in-vehicle
vision systems through contrast restoration, Proc. IEEE Conf.
Computer Vision, Pattern Recognition, pp. 1-8, June 2007.
[4] R. Fattal, Single image dehazing, ACM Transactions on
Graphics, vol. 27, no. 3, pp. 1-9, August 2008.
[5] R. Tan, Visibility in bad weather from a single image, Proc.
IEEE Conf. Computer Vision, Pattern Recognition, pp. 1-8, June
2008.
[6] M. Chen, A. Men, P. Fan, and B. Yang, Single image defogging,
IEEE Conference on Network Infrastructure and Digital Content,
pp. 675-679, November 2009.
[7] I. Yoon, J. Jeon, J. Lee, and J. Paik, Spatially adaptive image
defogging using edge analysis and gradient-based tone mapping,
Proc. IEEE Int. Conf. Consumer Electronics, pp. 195-196, January
2011.
[8] L. Kratz and K. Nishino, Factorizing scene albedo and depth
from a single foggy image, IEEE Int. Conf. Computer Vision, pp.
1701-1708, September 2009.
[9] N. Kong and H. Ibrahim, Color image enhancement using
brightness preserving dynamic histogram equalization, IEEE
Trans. Consumer Electronics, vol. 54, no. 4, pp. 1962-1968,
November 2008.
[10] Z. Xu, H. Wu, X. Yu, and B. Qiu, Colour image enhancement by
virtual histogram approach, IEEE Trans. Consumer Electronics,
vol. 56, no. 2, May 2010.
[11] K. Zhang, L. Zhang, and S. Zhang, A variational multiphase level
set approach to simultaneous segmentation and bias correction,
IEEE Int. Conf. Image Processing, pp. 4105-4108, December
2010.
[12] M. Ebner, Color constancy, John Wiley & Sons, England, 2007.

BIOGRAPHIES

Inhye Yoon was born in Suwon, Korea in 1988. She
received the B.S. degree in electronic engineering from
Kangnam University, Korea, in 2010. Currently, she is
pursuing M.S. degree in image processing at Chung-Ang
University. Her research interests include image
restoration, digital auto-focusing, image and video
processing, real-time object tracking.



Seonyoug Kim was born in Pohang, Korea in 1988. She
received the B.S. degree in information
telecommunications engineering from Suwon University,
Korea, in 2011. Currently, she is pursuing M.S. degree in
image processing at Chung-Ang University. Her research
interests include image restoration, digital auto-focusing,
image and video processing, real-time object tracking.



Donggyun Kim was born in Ulsan, Korea in 1983. He
received the B.S. and M. S. degree in electronic and
electrical engineering from Chung-Ang University, Korea,
in 2007 and 2009, respectively. Currently, he is pursuing
Ph.D. degree in image processing at Chung-Ang
University. His research interests include image
restoration, digital auto-focusing, image and video
processing, real-time object tracking.

116 IEEE Transactions on Consumer Electronics, Vol. 58, No. 1, February 2012
Monson Hayes received his BS Degree from the
University of California at Berkeley in 1971, worked as a
Systems Engineer at Aerojet Electrosystems until 1974,
and then received his Sc.D. degree in electrical
engineering and computer science from the Massachusetts
Institute of Technology in 1981. He then joined the
faculty at the Georgia Institute of Technology where he is
a Professor of Electrical and Computer Engineering. Dr. Hayes was a recipient
of the Presidential Young Investigator Award and the recipient of the IEEE
Senior Award. He has served the Signal Processing Society of the IEEE in
numerous positions, including Chairman of the DSP Technical Committee
(1995-1997), Associate Editor for the IEEE Transactions on Acoustics,
Speech, and Signal Processing (1984-1988) and Associate Editor for the IEEE
Transactions on Education (2000-2010), Secretary-Treasurer of the ASSP
Publications Board (1986-1988), and Chairman of the ASSP Publications
Board ((1992-1994), General Chairman of ICASSP 96, and General Chairman
of ICIP 2006. Currently, Dr. Hayes has served as an Associate Chair in the
School of ECE at Georgia Tech, and as Associate Director for Georgia Tech
Savannah. Since joining the faculty at Georgia Tech, Dr. Hayes has become
internationally recognized for his contributions to the field of digital signal
processing, image and video processing, and engineering education. He has
published over 150 papers, is the author of two textbooks, and has received
numerous awards and distinctions from professional societies and Georgia
Tech. His research interests are in the areas of face and gesture recognition,
image and video processing, adaptive signal processing, and engineering
education. In 1992 he was elected to the grade of Fellow of the IEEE. Dr.
Hayes is currently a Distinguished Foreign Professor at Chung-Ang
University, Seoul, Korea, in the Graduate School of Advanced Imaging
Science, Multimedia, and Film.











































Joonki Paik was born in Seoul, Korea in 1960. He
received the B.S. degree in control and instrumentation
engineering from Seoul National University in 1984. He
received the M.S. and the Ph.D. degrees in electrical
engineering and computer science from Northwestern
University in 1987 and 1990, respectively. From 1990 to
1993, he joined Samsung Electronics, where he designed
the image stabilization chip sets for consumers camcorders. Since 1993, he
has joined the faculty at Chung-Ang University, Seoul, Korea, where he is
currently a Professor in the Graduate school of Advanced Imaging Science,
Multimedia and Film. From 1999 to 2002, he was a visiting Professor at the
Department of Electrical and Computer Engineering at the University of
Tennessee, Knoxville. Dr. Paik was a recipient of Chester-Sall Award from
IEEE Consumer Electronics Society, Academic Award from the Institute of
Electronic Engineers of Korea, and Best Research Professor Award from
Chung-Ang University. He has served the Consumer Electronics Society of
IEEE as a member of the Editorial Board. Since 2005, he has been the head of
National Research Laboratory in the field of image processing and intelligent
systems. In 2008, he has worked as a full-time technical consultant for the
System LSI Division in Samsung Electronics, where he developed various
computational photographic techniques including an extended depth of field
(EDoF) system. From 2005 to 2007 he served as Dean of the Graduate School
of Advanced Imaging Science, Multimedia, and Film. From 2005 to 2007 he
has been Director of Seoul Future Contents Convergence (SFCC) Cluster
established by Seoul Research and Business Development (R&BD) Program.
Dr. Paik is currently serving as a member of Presidential Advisory Board for
Scientific/Technical policy of Korean Government and a technical consultant
of Korean Supreme Prosecutors Office for computational forensics.