Smoothing Techniques in Image

Processing
Prof. PhD. Vasile Gui
Polytechnic University
of Timisoara
Content
Introduction
Brief review of linear operators
Linear image smoothing techniques
Nonlinear image smoothing techniques
Introduction
Why do we need image
smoothing?

What is image and what
is noise?

Frequency spectrum
Statistical properties

Brief Review of Linear Operators
[Pratt 1991]
Generalized 2D linear operator


Separable linear operator:


Space invariant operator:




Convolution sum

=
=
1
0
1
0
) , ( ) , ; , ( ) , (
M
j
N
k
k j f n m k j O n m g
) ; ( ) ; ( ) , ; , ( n k O m j O n m k j O
C R
=

=
=
1
0
1
0
) , ( ) ; ( ) ; ( ) , (
M
j
N
k
C R
k j f n k O m j O n m g
) , ( ) , ( ) , ; , ( k n j m H k n j m O n m k j O = =

=
=
1
0
1
0
) , ( ) , ( ) , (
M
j
N
k
k j f k n j m h n m g
Brief Review of Linear Operators
Geometrical
interpretation
of 2D
convolution



h
0
0

h
01
h
02

h
10

h
20
h
21
h
22

h
11
h
12

h
0
0

h
01
h
02

h
10

h
20
h
21
h
22

h
11
h
12

h
0
0

h
01
h
02

h
10

h
20
h
21
h
22

h
11
h
12
h
0
0

h
01
h
02

h
10

h
20
h
21
h
22

h
11
h
12

N+L-1
N
N-
L+1
h
0
0

h
01
h
02

h
10

h
20
h
21
h
22

h
11
h
12

L-1
2
L-1
2
L-1
2
L-1
2 H
G
F
Matrix formulation

Unitary transform

Separable transform

Transform is invertible


Brief Review of Linear Operators
T
R C
FA A T =
If Af A t A f
*T *T
= = =
*T 1
A A =

Af t =
Brief Review of Linear Operators
Basis vectors are orthogonal


Inner product and energy are conserved


Unitary transform: a rotation in N-dimensional vector
space

=
=
= o =
l k
l k
l k
, 0
, 1
) , (
l
T *
k
a a
2 T * T * T * T *
*T *T 2
|| f || f f If f Af A f
(Af) (Af) t t || t ||
= = =
= = =
Brief Review of Linear Operators
2D vector space interpretation
of a unitary transform
x
1

x
2

2

Brief Review of Linear Operators
DFT unitary matrix












A =

(
(
(
(
(
(
(
(
(

1
0 0 0 0
0 1 2 1
0 2 4 2 1
0 2 1 1
2
N
W W W W
W W W W
W W W W
W W W W
N
N
N N N
( )
( ) ( )
)
2
sin( )
2
cos( }
2
exp{
N
i
N N
i
W
N
t

t
=
t
=
Brief Review of Linear Operators
1D DFT


2D DFT

1 ,..., 1 , 0 , }
2
exp{ ) (
1
) (
1
0
=
t
=

=
N u un
N
i
n f
N
u t
N
n
1 ,..., 1 , 0 , 1 ,..., 1 , 0
)} ( 2 exp{ ) , (
1
) , (
1
0
1
0
= =
+ t =

=
N v M u
n
N
v
m
M
u
i n m f
MN
v u t
M
m
N
n
Brief Review of Linear Operators
Circular convolution theorem




Periodicity of f,h and g with N are assumed
t u v
N
t u v t u v
g f h
( , ) ( , ) ( , ) =
1

=
=
1
0
1
0
) , ( ) , ( ) , (
N
j
N
k
k j f k n j m h n m g
Linear Image smoothing techniques
Box filters. Arithmetic mean LL operator


| |
(
(
(
(

=
(
(
(
(

=
1
1
1
1
* 1 1 1
1
1 1 1
1
1 1 1
1 1 1
1
2

L L L
h
Linear Image smoothing techniques
Box filters. Arithmetic mean LL operator
Separable
Can be computed recursively, resulting in
roughly 4 operations per pixel






L pixels
+

m,n

m,n+1

Linear Image smoothing techniques
Box filters. Arithmetic mean LL operator
Optimality properties
Signal and additive white noise



Noise variance is reduced N times
g f n f n = + = +
= = =

1 1 1
1 1 1
N N N
k k
k
N
k
k
N
k
k
N
( ) .
z n =
=

1
1
N
k
k
N
.
2
1 1
2
2
1 1
2
1 1
2
T 2
1
) , (
1
} E{
1
} E{
1
} E{
o o o
= = = = o

= =
= = = =
N
= k l
N
n n
N
n n
N
z z
N
k
N
l
N
k
N
l
k l
N
k
N
l
k l z
Linear Image smoothing techniques
Box filters. Arithmetic mean LL operator
Unknown constant signal plus noise
Minimize MSE of the estimation g:

=
= c
N
k
k
g f g
1
2 2
) ( ) (
0
) (
2
=
c
cc
g
g

=
=
N
k
k
f
N
g
1
1

Linear Image smoothing techniques

Box filters. Arithmetic mean LL operator
i.i.d. Gaussian signal with unknown mean.


Given the observed samples, maximize


Optimal solution: arithmetic mean
}
2
) (
exp{ . ) | (
2
2
o

=
f
ct f p
[
= =

o
=
N
k
N
k
k k
f ct f p
1 1
2
2
} ) (
2
1
exp{ . ) | (
Linear Image smoothing techniques
Box filters. Arithmetic mean LL operator
Frequency response:
Non-monotonically decreasing with frequency

1
9
1 1 1
1 1 1
1 1 1
1
3
111
1
3
1
1
1

(
(
(
= -

(
(
(
= - [ ] h h
x y
t u h h n
i
N
un
x x
n N
N
( ) ( ) ( ) exp{ }
( )/
( )/
= +
=

0
2
1 2
1 2
t
)
2
cos(
3
2
3
1
) ( u
N
u t
t
+ =
Linear Image smoothing techniques
Box filters. Arithmetic mean LL operator
An example










8 5 8 8 5 8
8 5 8 8 5 8
8 5 8 8 5 8
1
9
1 1 1
1 1 1
1 1 1
7 7 7 7 7 7
7 7 7 7 7 7
7 7 7 7 7 7
-

(
(
(
=










8 5 8 5 8 5
8 5 8 5 8 5
8 5 8 5 8 5
1
9
1 1 1
1 1 1
1 1 1
6 7 6 7 6 7
6 7 6 7 6 7
6 7 6 7 6 7
-

(
(
(
=
Linear Image smoothing techniques
Box filters. Arithmetic mean LL operator
Image smoothed with
33, 55, 99
and 11 11 box filters
Linear Image smoothing techniques
Box filters. Arithmetic mean LL operator
Original Lena image
Lena image filtered with
5x5 box filter
Linear Image smoothing techniques
Binomial filters [Jahne 1995]
Computes a weighted average of pixels in
the window
Less blurring, less noise cleaning for the
same size
The family of binomial filters can be defined
recursively
The coefficients can be found from (1+x)
n
Linear Image smoothing techniques
Binomial filters. 1D versions
| | 1 1
2
1
=
1
b
| |
1 1
b b b * 1 2 1
4
1
2
= =
| |
1 1 1 1
b b b b b * * * 1 4 6 4 1
16
1
4
= =
| |
1 1 1 1 1 1
b b b b b b b * * * * * 1 6 15 20 15 6 1
64
1
6
= =
As size increases, the shape of the filter is closer to a Gaussian one
Linear Image smoothing techniques
Binomial filters. 2D versions
| |
(
(
(

=
(
(
(

=
1 2 1
2 4 2
1 2 1
16
1
1
2
1
4
1
* 1 2 1
4
1
2
b
| |
(
(
(
(
(
(

=
(
(
(
(
(
(

=
1 4 6 4 1
4 16 24 16 4
6 24 36 24 6
4 16 24 16 4
1 4 6 4 1
256
1
1
4
6
4
1
16
1
* 1 4 6 4 1
16
1
4
b
Linear Image smoothing techniques
Binomial filters. Frequency response
| | 1 2 1
4
1
2

= b
)
2
cos(
2
1
2
1
) ( u
N
u t
t
+ =
| | 1 4 6 4 1
16
1
4

= b
2
)]
2
cos(
2
1
2
1
[ ) ( u
N
u t
t
+ =
Monotonically decreasing
with frequency
Linear Image smoothing techniques
Binomial filters. Example
Original Lena image
Lena image filtered
with binomial 5x5 kernel
Lena image filtered
with box filter 5x5
Linear Image smoothing techniques
Binomial and box filters. Edge blurring comparison
Linear filters have to compromise smoothing
with edge blurring
Step edge
Result of size L
box filter

L
Size L binomial
Nonlinear image smoothing
The median filter [Pratt 1991]
Block diagram
2
1 +
=
N
m
f
1

f
2

f
N

.
.
.
Order
samples
f
(1)

f
(2)

f
(N)

.
.
.
select
f
(m)

median
N is odd
) ( ) 3 ( ) 2 ( ) 1 ( N
f f f f s s s
Nonlinear image smoothing
The median filter
Numerical example
6
7
3 7 8
2 3
4 6 7
2, 3, 3, 4, 6, 7, 7, 7, 8
Nonlinearity
Nonlinear image smoothing
The median filter
median{ f1 + f2} = median{ f1} + median{ f2}.
However:
median{ c f } = c median{ f },
median{ c + f } = c + median{ f }.
The filter selects a sample from the window, does
not average
Edges are better preserved than with liner filters
Best suited for salt and pepper noise
Nonlinear image smoothing
The median filter
Noisy image 5x5 median filtered 5x5 box filter
Nonlinear image smoothing
The median filter
Optimality
Grey level plateau plus noise. Minimize sum of
absolute differences:


Result:

If 51% of samples are correct and 49% outliers,
the median still finds the right level!

=
= c
N
k
k
g f g
1
| | ) (
) ( 3 2 1
} , , , , {
m N
f f f f f median g = =
Nonlinear image smoothing
The median filter
Caution: points, thin lines and corners are erased by the median filter
Test images
Results of 33 pixel median filter
Nonlinear image smoothing
The median filter
Cross shaped window can correct some of the problems
Results of the 9 pixel cross shaped window median filter
Nonlinear image smoothing
The median filter
Implementing the median filter
Sorting needs O(N
2
) comparisons
Bubble sort
Quick sort
Huang algorithm (based on histogram)
VLSI median

Nonlinear image smoothing
The median filter
VLSI median block diagram
a
b
Min(a,b)
Max(a,b)
x
(1)

x
(2)

x
(3)

x
(4)

x
(5)

x
(6)

x
(7)

x
1

x
2

x
3

x
4

x
5

x
6

x
7

Nonlinear image smoothing
The median filter
Color median filter
There is no natural ordering in 3D (RGB) color
space
Separate filtering on R,G and B components does
not guarantee that the median selects a true
sample from the input window
Vector median filter, defined as the sample
minimizing the sum of absolute deviations from all
the samples
Computing the vector median is very time
consuming, although several fast algorithms exist
Nonlinear image smoothing
The median filter
Example of color
median filtering
5x5 pixels window
Up: original image
Down: filtered image
Nonlinear image smoothing
The weighted median filter
The basic idea is to give higher weight to some samples, according to
their position with respect to the center of the window
Each sample is given a weight according to its spatial position in the
window.
Weights are defined by a weighting mask
Weighted samples are ordered as usually
The weighted median is the sample in the ordered array such that
neither all smaller samples nor all higher samples can cumulate more
than 50% of weights.
If weights are integers, they specify how many times a sample is
replicated in the ordered array
Nonlinear image smoothing
The weighted median filter
Numerical example for the weighted median filter
6
7
3 7 8
2 3
4 6 7
2, 2, 3, 3, 3, 3, 4, 6, 6, 7, 7, 7, 7, 7, 8

3

1

1

1

1

2

2

2

2
Nonlinear image smoothing
The multi-stage median filter
Better detail preservation
m
i
= median( R
i
), i =1,2,3,4.
Result = median( m
1
,m
2
,m
3
,m
4
,m
5
)
R
1

R
2

R
3

R
4

m
5

- - -
- - -
- - - - -
- - -
- - -
Nonlinear image smoothing
Rank-order filters (L filters)
Block diagram
E sum
y
f
(1)

f
1

f
2

f
N

.
.
.
ordering
f
(2)

f
(N)

.
.
.
-a
1

-a
2

-a
N

weights
y = a
1
f
(1)
+ a
2
f
(2)
+...+ a
N
f
(N)

a
k
k
N
=

=
1
1
Nonlinear image smoothing
Rank-order filters (L filters)
Some examples
a
1
=1
Min
. . .
. . .
a
N
=1
Max
. . . . . .
a
m
=1
Median
a
1
=.5 a
N
=.5
Mid range
. . .
Particular cases
of grey level
morphological
filters
Percentile
filters (rank
selection):
0%,
50%
100%
Nonlinear image smoothing
Rank-order filters (L filters)
Optimality

c(

) |

|
( )
f f f
k
r
k
N
=
=

1
Minkovski distance
Case r = 1: best estimator is median.
Case r = 2, best estimator is arithmetic mean.
Case r , best estimator is mid-range.
Nonlinear image smoothing
Rank-order filters (L filters)
Alpha-trimmed mean filter

+ s s +
=
otherwise
Q m k Q m for Q
a
k
, 0
), 1 2 /( 1
o = (2Q + 1) / N, defines de degree of averaging
o= 1 corresponds to arithmetic mean
o = 1 / N corresponds to the median filter
Properties: in between mean and median
Nonlinear image smoothing
Rank-order filters (L filters)
Median of absolute differences trimmed mean
Better smoothing than the median filter and good edge
preservation

} , , , , {
3 2 1 1 N
f f f f median M =
N i M f median M
i
,..., 2 , 1 |}, {|
1 2
= =
} | | : {
2 1
M M f f average output
i i
s =
M
2
is a robust estimator of the variance
Nonlinear image smoothing
Rank-order filters (L filters)
k nearest neighbour (kNN) median filter
Median of the k grey values nearest by rank
to the central pixel
Aim: same as above
Kuwahara type filtering
Form regions R
i
Compute mean, m
i
and variance,
S
i
for each region.
Result = m
i
: m
i
< m
j
for all j
different from i, i,e. the mean of
the most homogeneous region
Matsujama & Nagao
Nonlinear image smoothing
Selected area filtering [Nagao 1980]
reg 1
reg 2
reg 3
reg 4
reg 5
reg 6
reg 7 reg 8
Nonlinear image smoothing
Conditional mean
Pixels in a neighbourhood are averaged only if they
differ from the central pixel by less than a given
threshold:

<
=
=

= =
otherwise
th n m f l n k m f if
l k h
l n k m f l k h n m g
L
L k
L
L l
, 0
| ) , ( ) , ( | , 1
) , (
), , ( ) , ( ) , (
L is a space scale parameter and th is a range scale parameter
Nonlinear image smoothing
Conditional mean
Example with L=3,
th=32
Nonlinear image smoothing
Bilateral filter [Tomasi 1998]
Space and range are treated in a similar way
Space and range similarity is required for the averaged pixels
Tomasi and Manduchi [1998] introduced soft weights to penalize the
space and range dissimilarity.
)) , ( ) , ( ( ) , ( ) , ( n m f l n k m f r l k s l k h =

=
=
k l
k l
l k h K
l n k m f l k h
K
n m g
) , (
, ) , ( ) , (
1
) , (
s() and r() are space and range similarity functions (Gaussian
functions of the Euclidian distance between their arguments).
Nonlinear image smoothing
Bilateral filter
The filter can be seen as weighted averaging in the
joint space-range space (3D for monochromatic
images and 5D x,y,R,G,B - for colour images)
The vector components are supposed to be properly
normalized (divide by variance for example)
The weights are given by:
) ); K(d( ) (
}
|| ||
exp{ ) (
2
s h
s
h
c
c
x x x
x x
x
=

=
Nonlinear image smoothing
Bilateral filter
Example of Bilateral filtering
Low contrast texture has been removed
Yet edges are well preserved
Nonlinear image smoothing
Mean shift filtering [Comaniciu 1999, 2002]
Mean shift filtering replaces each pixels value with the most probable local value, found
by a nonparametric probability density estimation method.



The multivariate kernel density estimate obtained in the point x with the kernel K(x) and
window radius r is:



For the Epanechnikov kernel, the estimated normalized density gradient is proportional
to the mean shift:
d
i n i i
R e
=
x x
... 1
} {

=
|
.
|

\
|

=
n
i
d
r
K
nr
f
1
1
) (

i
x x
x

e
= =
V
+
) (
2
1
) (
) (

) (

2
x x
i
x
x x x
x
x
r i
S
r
n
M
f
f
d
r
S is a sphere of radius r, centered on x and n
x
is the number of samples inside the sphere
Nonlinear image smoothing
Mean shift filtering
The mean shift procedure is
a gradient ascent method to
find local modes (maxima)
of the probability density
and is guaranteed to
converge.
Step1: computation of the
mean shift vector M
r
(x).
Step2: translation of the
window S
r
(x) by M
r
(x).
Iterations start for each pixel
(5D point) and tipically
converge in 2-3 steps.
Nonlinear image smoothing
Mean shift filtering
Example1.
Nonlinear image smoothing
Mean shift filtering
Detail of a 24x40
window from the
cameraman image
a) Original data
b) Mean shift paths for
some points
c) Filtered data
d) Segmented data

Nonlinear image smoothing
Mean shift filtering
Example 2
Nonlinear image smoothing
Mean shift filtering
Comparison to bilateral filtering
Both methods based on simultaneous processing of both the
spatial and range domains
While the bilateral filtering uses a static window, the mean shift
window is dynamic, moving in the direction of the maximum
increase of the density gradient.
REFERENCES
D. Comaniciu, P. Meer: Mean shift analysis and applications. 7th
International Conference on Computer Vision, Kerkyra, Greece, Sept.
1999, 1197-1203.
D. Comaniciu, P. Meer: Mean shift: a robust approach toward feature
space analysis. IEEE Trans. on PAMI Vol. 24, No. 5, May 2002, 1-18.
M. Elad: On the origin of bilateral filter and ways to improve it. IEEE
trans. on Image Processing Vol. 11, No. 10, October 2002, 1141-1151.
B. Jahne: Digital image processing. Springer Verlag, Berlin 1995.
M. Nagao, T. Matsuiama: A structural analysis of complex aerial
photographs. Plenum Press, New York, 1980.
W.K. Pratt: Digital image processing. John Wiley and sons, New York
1991
C. Tomasi, R. Manduchi: Bilateral filtering for gray and color images.
Proc. Sixth Intl. Conf. Computer Vision , Bombay, 839-846.