Smoothing Techniques in Image

Processing

Prof. PhD. Vasile Gui

Polytechnic University
of Timisoara
Content

Introduction
Briefreview of linear operators
Linear image smoothing techniques
Nonlinear image smoothing techniques
Introduction

Why do we need image

smoothing?

What is image and what

is noise?

Frequency spectrum
Statistical properties
Brief Review of Linear Operators
[Pratt 1991]

Generalized 2D linear operator

M 1 N 1
g (m, n) O( j, k ; m, n) f ( j, k )
j 0 k 0

Separable linear operator:

O( j , k ; m, n) O R ( j; m)OC (k ; n)
M 1 N 1
g ( m, n) O
j 0
R ( j ; m) OC ( k ; n) f ( j , k )
k 0
Space invariant operator:
O( j , k ; m, n) O(m j , n k ) H (m j , n k )
M 1 N 1
g ( m, n) h( m
j 0 k 0
j, n k ) f ( j, k )

Convolution sum
Brief Review of Linear Operators
h22 h21 h20
L-1
h12 h11 h10 2
H h02 h01 h0
L-1
2 h22 h21 h20 Geometrical
0
h12 h11 h10

G
L-1
2
L-1 h22 h21 h20
h02 h01 h0
0
interpretation
h12 h11 h10
2
h02 h01 h0 of 2D
F
0
N-
L+1
convolution
N

N+L-1

h22 h21 h20

h12 h11 h10 h22 h21 h20
h02 h01 h0 h12 h11 h10
0
h02 h01 h0
Brief Review of Linear Operators

Matrix formulation
t Af
Unitary transform
1
A A *T

Separable transform
T A C FA T
R
Transform is invertible
f A *T t A *T Af If
Brief Review of Linear Operators

Basis vectors are orthogonal

1, k l
a k a l ( k , l )
*T

0, k l
Inner product and energy are conserved
|| t || 2 t *T t (Af)*T (Af)
f *T A *T Af f *T If f *T f || f || 2
Unitary transform: a rotation in N-dimensional vector
space
Brief Review of Linear Operators

x2

2 1

x1

2D vector space interpretation

of a unitary transform
Brief Review of Linear Operators

DFT unitary matrix

W0 W0 W0 W0

W 0
W1 W2 W N 1
W0 W2 W4 W 2 ( N 1)
1
A
N



W 0 W N W 2 ( N 1) W ( N 1) 2

i 2 2 2
W N exp{ } cos( ) i sin( )
N N N
Brief Review of Linear Operators

1D DFT
1 N 1
i 2
t (u )
N
n 0
f (n) exp{
N
un}, u 0,1,..., N 1

2D DFT M 1 N 1
1 u v
t (u , v)
MN m 0 n 0
f (m, n) exp{i 2( m n)}
M N
u 0,1,..., M 1, v 0,1,..., N 1
Brief Review of Linear Operators

Circular convolution theorem

N 1 N 1
g (m, n) h(m j , n k ) f ( j , k )
j 0 k 0
1
t g (u, v ) t f ( u, v )t h ( u, v )
N
Periodicity of f,h and g with N are assumed
Linear Image smoothing techniques
Box filters. Arithmetic mean LL operator

1 1 1 1
1 1 1 1
1 1 1
h 2 1 1 1 *
L 1 L L

1 1 1 1
Linear Image smoothing techniques
Box filters. Arithmetic mean LL operator

Separable
Can be computed recursively, resulting in
roughly 4 operations per pixel
m,n

m,n+1

L pixels
Linear Image smoothing techniques
Box filters. Arithmetic mean LL operator

Optimality properties
Signal and additive white noise
N N N
1 1 1
g
N

k 1
( f k nk )
N

k 1
fk
N
n .
k 1
k

N
1
z
N
n .
k 1
k

Noise varianceN is reducedN NN times

N
1 1
E{z z} 2 E{ nl n k } 2
2
z
T
E{n n } l k
N k 1 l 1 N k 1 l 1

1 N N 1 2
2
(l , k ) 2
=
N k 1 l 1 N
Linear Image smoothing techniques
Box filters. Arithmetic mean LL operator

Unknown constant signal plus noise

Minimize MSE of the estimation g:
N
(g) ( fk g)2
2

k 1

2 ( g )
0
g
N
1
g
N
f
k 1
k
Linear Image smoothing techniques
Box filters. Arithmetic mean LL operator

i.i.d. Gaussian signal with unknown mean.

( f ) 2
p ( f | ) ct. exp{ }
2 2

Given the observed samples, maximize

N N
1

k 1
p ( f k | ) ct . exp{
2 2
( f
k 1
k ) 2 }

Optimal solution: arithmetic mean

Linear Image smoothing techniques
Box filters. Arithmetic mean LL operator

Frequency response:
Non-monotonically decreasing with frequency

1 1 1 1
1 1 1
1 1 1 [111] 1 hx hy
9 3 3
1 1 1 1
( N 1)/ 2
i 2
t (u) hx (0)
n ( N 1)/ 2
hx (n) exp{
N
un}

1 2 2
t (u ) cos( u )
3 3 N
Linear Image smoothing techniques
Box filters. Arithmetic mean LL operator

An example

8 5 8 8 5 8 1 1 1 7 7 7 7 7 7
1
8 5 8 8 5 8 1 1 1 7 7 7 7 7 7
9
8 5 8 8 5 8 1 1 1 7 7 7 7 7 7

8 5 8 5 8 5 1 1 1 6 7 6 7 6 7
1
8 5 8 5 8 5 1 1 1 6 7 6 7 6 7
9
8 5 8 5 8 5 1 1 1 6 7 6 7 6 7

Linear Image smoothing techniques
Box filters. Arithmetic mean LL operator

Image smoothed with

33, 55, 99
and 11 11 box filters
Linear Image smoothing techniques
Box filters. Arithmetic mean LL operator

Lena image filtered with

Original Lena image
5x5 box filter
Linear Image smoothing techniques
Binomial filters [Jahne 1995]

Computes a weighted average of pixels in

the window
Less blurring, less noise cleaning for the
same size
The family of binomial filters can be defined
recursively
The coefficients can be found from (1+x) n
Linear Image smoothing techniques
Binomial filters. 1D versions

1
b1 1 1
2
1
b 1 2 1 b 1 * b 1
2

4
1
b 1 4 6 4 1 b 1 * b 1 * b 1 * b 1
4

16
1
b 1 6 15 20 15 6 1 b 1 * b 1 * b 1 * b 1 * b 1 * b 1
6

64
As size increases, the shape of the filter is closer to a Gaussian one
Linear Image smoothing techniques
Binomial filters. 2D versions

1 1 2 1
1 1 1
b 2 1 2 1 * 2 2 4 2
4 4 16
1 1 2 1

1 1 4 6 4 1
4 4 16 24 16 4
1 1 1
b 1 4 6 4 1 * 6
4
6 24 36 24 6
16 16 256
4 4 16 24 16 4
1 1 4 6 4 1
Linear Image smoothing techniques
Binomial filters. Frequency response

1

b 1 2 1
2

4
t (u )
1 1 2
cos( u )
2 2 N

b4
1
16
1 4 6 4 1

1 1
2 2
2
t (u ) [ cos( u )]2
N

Monotonically decreasing
with frequency
Linear Image smoothing techniques
Binomial filters. Example

Original Lena image Lena image filtered Lena image filtered

with binomial 5x5 kernel with box filter 5x5
Linear Image smoothing techniques
Binomial and box filters. Edge blurring comparison

Linear filters have to compromise smoothing

with edge blurring
Step edge

Result of size L
box filter

Size L binomial

L
Nonlinear image smoothing
The median filter [Pratt 1991]

Block diagram
f(1)
f1
f(2)
f2

. . f(m)
. Order . select
.
. samples
f(N) median
fN

f (1) f ( 2) f ( 3) f ( N )

N 1
m N is odd
2
Nonlinear image smoothing
The median filter

Numerical example

3 7 8
2 3 7 6
4 6 7

2, 3, 3, 4, 6, 7, 7, 7, 8
Nonlinear image smoothing
The median filter

Nonlinearity

median{ f1 + f2} median{ f1} + median{ f2}.

However:
median{ c f } = c median{ f },
median{ c + f } = c + median{ f }.

The filter selects a sample from the window, does

not average
Edges are better preserved than with liner filters
Best suited for salt and pepper noise
Nonlinear image smoothing
The median filter

Noisy image 5x5 median filtered 5x5 box filter

Nonlinear image smoothing
The median filter

Optimality
Grey level plateau plus noise. Minimize sum of
absolute differences:
N
( g ) | f k g |
k 1
Result:
g median{ f 1 , f 2 , f 3 , , f N } f ( m )
If 51% of samples are correct and 49% outliers,
the median still finds the right level!
Nonlinear image smoothing
The median filter

Caution: points, thin lines and corners are erased by the median filter

Test images

Results of 33 pixel median filter

Nonlinear image smoothing
The median filter

Cross shaped window can correct some of the problems

Results of the 9 pixel cross shaped window median filter

Nonlinear image smoothing
The median filter

Implementing the median filter

Sorting needs O(N2) comparisons
Bubble sort
Quick sort
Huang algorithm (based on histogram)
VLSI median
Nonlinear image smoothing
The median filter

VLSI median block diagram

x1 x(1)

x2 x(2)

x3 x(3)
a Min(a,b)
x4 x(4)
b Max(a,b)
x5 x(5)

x6 x(6)

x7 x(7)
Nonlinear image smoothing
The median filter

Color median filter

There is no natural ordering in 3D (RGB) color
space
Separate filtering on R,G and B components does
not guarantee that the median selects a true
sample from the input window
Vector median filter, defined as the sample
minimizing the sum of absolute deviations from all
the samples
Computing the vector median is very time
consuming, although several fast algorithms exist
Nonlinear image smoothing
The median filter

Example of color
median filtering
5x5 pixels window
Up: original image
Down: filtered image
Nonlinear image smoothing
The weighted median filter

The basic idea is to give higher weight to some samples, according to

their position with respect to the center of the window
Each sample is given a weight according to its spatial position in the
window.
Weights are defined by a weighting mask
Weighted samples are ordered as usually
The weighted median is the sample in the ordered array such that
neither all smaller samples nor all higher samples can cumulate more
than 50% of weights.
If weights are integers, they specify how many times a sample is
replicated in the ordered array
Nonlinear image smoothing
The weighted median filter

Numerical example for the weighted median filter

1 2 1

2 3 2

1 2 1

3 7 8
2 3 7 6
4 6 7

2, 2, 3, 3, 3, 3, 4, 6, 6, 7, 7, 7, 7, 7, 8
Nonlinear image smoothing
The multi-stage median filter

Better detail preservation

R3

R2 R4

m5
R1

mi = median( Ri ), i =1,2,3,4.

Result = median( m1,m2,m3,m4,m5 )

Nonlinear image smoothing
Rank-order filters (L filters)

Block diagram
weights

f(1)
f1 a1
f(2)
f2
a2
y
. .
.
ordering . sum
. .

f(N)
fN
aN

a
k 1
k 1 y = a1f(1) + a2 f(2)+...+ aN f(N)
Nonlinear image smoothing
Rank-order filters (L filters)

Some examples
a1=1

Percentile . . .
Min
Particular cases
filters (rank of grey level
selection): aN=1 morphological
filters
. . . Max
0%,
50% am=1

. . . . . .
100% Median

a1=.5 aN=.5
. . .
Mid range
Nonlinear image smoothing
Rank-order filters (L filters)

Optimality
N
( f ) | f ( k ) f |r Minkovski distance
k 1

Case r = 1: best estimator is median.

Case r = 2, best estimator is arithmetic mean.
Case r , best estimator is mid-range.
Nonlinear image smoothing
Rank-order filters (L filters)

Alpha-trimmed mean filter

1 /(2Q 1), for m Q k m Q
ak
0, otherwise

= (2Q + 1) / N, defines de degree of averaging

= 1 corresponds to arithmetic mean
= 1 / N corresponds to the median filter

Properties: in between mean and median

Nonlinear image smoothing
Rank-order filters (L filters)

Median of absolute differences trimmed mean

Better smoothing than the median filter and good edge preservation

M 1 median{ f1 , f 2 , f 3 , , f N }
M 2 median{| f i M 1 |}, i 1,2,..., N
output average{ f i : | f i M 1 | M 2 }

M2 is a robust estimator of the variance

Nonlinear image smoothing
Rank-order filters (L filters)

k nearest neighbour (kNN) median filter

Median of the k grey values nearest by rank
to the central pixel
Aim: same as above
Nonlinear image smoothing
Selected area filtering [Nagao 1980]

Kuwahara type filtering

Form regions Ri
Compute mean, mi and variance, reg 1 reg 2
Si for each region.
reg 4
Result = mi : mi < mj for all j
different from i, i,e. the mean of reg 3
the most homogeneous region
Matsujama & Nagao reg 6
reg 5

reg 8 reg 7
Nonlinear image smoothing
Conditional mean

Pixels in a neighbourhood are averaged only if they

differ from the central pixel by less than a given
threshold:
L L
g (m, n) h(k , l ) f (m k , n l ),
k L l L

1, if | f (m k , n l ) f (m, n) | th
h( k , l )
0, otherwise

L is a space scale parameter and th is a range scale parameter

Nonlinear image smoothing
Conditional mean

Example with L=3,

th=32
Nonlinear image smoothing
Bilateral filter [Tomasi 1998]

Space and range are treated in a similar way

Space and range similarity is required for the averaged pixels
Tomasi and Manduchi [1998] introduced soft weights to penalize the
space and range dissimilarity.

h(k , l ) s (k , l )r ( f (m k , n l ) f (m, n))

1
g ( m, n)
K k l
h( k , l ) f ( m k , n l ),

K h( k , l )
k l

s() and r() are space and range similarity functions (Gaussian
functions of the Euclidian distance between their arguments).
Nonlinear image smoothing
Bilateral filter

The filter can be seen as weighted averaging in the

joint space-range space (3D for monochromatic
images and 5D x,y,R,G,B - for colour images)
The vector components are supposed to be properly
normalized (divide by variance for example)
The weights are given by:

|| x c x || 2
h(x) exp{ }
s
h(x) K(d(x c x); s )
Nonlinear image smoothing
Bilateral filter

Example of Bilateral filtering

Low contrast texture has been removed
Yet edges are well preserved
Nonlinear image smoothing
Mean shift filtering [Comaniciu 1999, 2002]

Mean shift filtering replaces each pixels value with the most probable local value, found
by a nonparametric probability density estimation method.

{x i }i 1...n xi R d
The multivariate kernel density estimate obtained in the point x with the kernel K(x) and
window radius r is:
1 n
x xi
f (x) d
nr
K
i 1 r


For the Epanechnikov kernel, the estimated normalized density gradient is proportional
to the mean shift:
r2 f ( x) 1
d 2 f (x)
M r ( x)
nx
x
x i S r ( x )
i x

S is a sphere of radius r, centered on x and nx is the number of samples inside the sphere
Nonlinear image smoothing
Mean shift filtering

The mean shift procedure is

a gradient ascent method to
find local modes (maxima)
of the probability density and
is guaranteed to converge.
Step1: computation of the
mean shift vector Mr(x).
Step2: translation of the
window Sr(x) by Mr(x).
Iterations start for each pixel
(5D point) and tipically
converge in 2-3 steps.
Nonlinear image smoothing
Mean shift filtering

Example1.
Nonlinear image smoothing
Mean shift filtering

Detail of a 24x40
window from the
cameraman
image
a) Original data
b) Mean shift paths for
some points
c) Filtered data
d) Segmented data
Nonlinear image smoothing
Mean shift filtering

Example 2
Nonlinear image smoothing
Mean shift filtering

Comparison to bilateral filtering

Both methods based on simultaneous processing of both the
spatial and range domains
While the bilateral filtering uses a static window, the mean shift
window is dynamic, moving in the direction of the maximum
increase of the density gradient.
REFERENCES
D. Comaniciu, P. Meer: Mean shift analysis and applications. 7th
International Conference on Computer Vision, Kerkyra, Greece, Sept.
1999, 1197-1203.
D. Comaniciu, P. Meer: Mean shift: a robust approach toward feature
space analysis. IEEE Trans. on PAMI Vol. 24, No. 5, May 2002, 1-18.
M. Elad: On the origin of bilateral filter and ways to improve it. IEEE
trans. on Image Processing Vol. 11, No. 10, October 2002, 1141-1151.
B. Jahne: Digital image processing. Springer Verlag, Berlin 1995.
M. Nagao, T. Matsuiama: A structural analysis of complex aerial
photographs. Plenum Press, New York, 1980.
W.K. Pratt: Digital image processing. John Wiley and sons, New York
1991
C. Tomasi, R. Manduchi: Bilateral filtering for gray and color images.
Proc. Sixth Intl. Conf. Computer Vision , Bombay, 839-846.