You are on page 1of 53

Computer Vision -

Restoration
Hanyang University

Jong-Il Park

Department of Computer Science and Engineering, Hanyang University
Restoration vs. Enhancement
Restoration
Objective process

A priori knowledge on
degradation model


Modeling the degradation and
applying the inverse process
to recover the original
To improve an image in some predefined sense
Enhancement
Subjective process

Department of Computer Science and Engineering, Hanyang University
Restoration process

Department of Computer Science and Engineering, Hanyang University
Noise models
Assume noise is independent
of spatial coordinates and it is
uncorrelated w.r.t. the image.
Gaussian: electronic circuit noise,
sensor noise
Rayleigh: range images
Exponential and gamma:
laser images
impulse(salt-and-pepper):
faulty switching


Department of Computer Science and Engineering, Hanyang University
Eg. Sample noisy images

Department of Computer Science and Engineering, Hanyang University
Eg. Sample noisy images(cont.)

Department of Computer Science and Engineering, Hanyang University
Periodic noise
Spatially dependent noise
Periodic noise can be
reduced significantly via
frequency domain filtering

Department of Computer Science and Engineering, Hanyang University
Estimation of noise parameters
PDF from small patches

Department of Computer Science and Engineering, Hanyang University
When the only degradation is noise
Periodic noise subtraction gives a good result
Random noise mean filter, order-statistics filter,
) , ( ) , ( ) , (
and
) , ( ) , ( ) , (
v u N v u F v u G
y x y x f y x g
+ =
+ = q

Department of Computer Science and Engineering, Hanyang University
Mean filters
Arithmetic mean filters
For Gaussian or uniform noise
Geometric mean filters
For Gaussian or uniform noise
Harmonic mean filters
Work well for salt noise but fail for pepper noise
Contraharmonic mean filters
Suited for impulse noise but require identification(salt
or pepper)

Department of Computer Science and Engineering, Hanyang University
Arithmetic & Geometric mean filter

Department of Computer Science and Engineering, Hanyang University
Contraharmonic filters
Q<0 : eliminates salt noise
Q=-1 harmonic mean filter
Q=0 : arithmetic mean filter
Q>0: eliminates pepper noise

e
e
+
=
xy
xy
S t s
Q
S t s
Q
t s g
t s g
y x f
) , (
) , (
1
) , (
) , (
) , (


Department of Computer Science and Engineering, Hanyang University
Eg. Contraharmonic filters

Department of Computer Science and Engineering, Hanyang University
Wrong sign in contraharmonic filters
Disaster!

Department of Computer Science and Engineering, Hanyang University
Order-Statistics filters
Median filter
Max filter
Min filter
Midpoint filter
Alpha-trimmed mean filter

Department of Computer Science and Engineering, Hanyang University
Median filters
3x3
median
3x3
median
3x3
median
blurred

Department of Computer Science and Engineering, Hanyang University
Max and Min filter
Max filter Min filter
Removes pepper noise
Removes dark pixels
Removes salt noise
Removes light pixels
Makes dark objects larger

Department of Computer Science and Engineering, Hanyang University
Eg. Comparison
(a) Additive uniform
noise
(b) (a)+additive S&P
5x5 arithmetic
mean
5x5 geometric mean
5x5 median 5x5 alpha-trimmed
Mean(d=5)

Department of Computer Science and Engineering, Hanyang University
Adaptive filters
Behavior changes locally based on statistical
characteristics of local support

Simple adaptive filter based on mean and variance
1. If global_var is zero, then f(x,y)=g(x,y)
2. If local_var>global_var, then f(x,y)=g(x,y) (high local
var edge should be preserved)
3. If local_var==global_var, then arithmetic mean filtering


Department of Computer Science and Engineering, Hanyang University
Eg. Adaptive filter

Department of Computer Science and Engineering, Hanyang University
Adaptive median filter
Cope with impulse noise with large probability
Preserve detail while smoothing non-impulse noise
Level A: A1=z
med
-z
min
A2=z
med
-z
max
If A1>0 AND A2<0, go to level B
Else increase the window size
If window size<=S
max
repeat level A
Else output z
xy

Level B: B1=z
xy
-z
min
B2=z
xy
-z
max
If B1>0 AND B2<0, output z
xy
Else output z
med
Algorithm

Department of Computer Science and Engineering, Hanyang University
Eg. Adaptive median filter
median adaptive median

Department of Computer Science and Engineering, Hanyang University
Periodic noise reduction
By frequency domain filtering
Band reject filter

Department of Computer Science and Engineering, Hanyang University
Eg. Periodic noise reduction

Department of Computer Science and Engineering, Hanyang University
Noise extraction
By bandpass filter
Help understanding noise pattern

Department of Computer Science and Engineering, Hanyang University
Notch filters

Department of Computer Science and Engineering, Hanyang University
Eg. Notch filtering
Removing sensor scan-
line patterns

Department of Computer Science and Engineering, Hanyang University
Optimum notch filtering
First isolating the principal contributions of the
interference pattern
Then subtracting weighted portion of the pattern from
the corrupted image

Department of Computer Science and Engineering, Hanyang University
Eg. Periodic interference(1/3)
Noisy image

Department of Computer Science and Engineering, Hanyang University
Eg. Periodic interference(2/3)
Extraction of noise interference pattern

Department of Computer Science and Engineering, Hanyang University
Eg. Periodic interference(3/3)
Restored image by subtracting weighted portion of
periodic interference (Refer to the derivation of
weights in pp.250-252)


Department of Computer Science and Engineering, Hanyang University
Linear, Position-Invariant Degradation
) , ( ) , ( ) , ( ) , (
: Domain Frequency
) , ( ) , ( ) , ( ) , (
: Domain Spatial
v u N v u F v u H v u G
y x y x f y x h y x g
+ =
+ - = q

Department of Computer Science and Engineering, Hanyang University
Degradation knowledge
Degradation knowledge about
1. A priori (known)
2. A posteriori (unknown)


blind restoration
or blind deconvolution
f
g
H
Restoration:
determine the original image ,
given the observed image and
knowledge about the degradation (H).

Department of Computer Science and Engineering, Hanyang University
Fundamental issue
Restoration problem

restoration is to find , such that
but, 1. does not exist: singular
2. may exist, but not be unique: ill-conditioned
3. may exist and unique, but there exists ,
which can be made arbitrarily small, such that
which is not negligible

Image restoration is ill-conditioned at best and
singular at worst

g f T = } {
1
T
f g T =

} {
1
1
T
1
T
1
T c
, , } {
1
c o o c >> + = +

f g T

Department of Computer Science and Engineering, Hanyang University
Estimation of degradation function
Approaches

Observation

Experimentation

Mathematical modeling


Department of Computer Science and Engineering, Hanyang University
Estimation by observation
Looking at a small section of the image containing
simple structures and then obtaining degradation
function

Observed sub-image:
Estimate of original image:

) , ( y x g
s
) , (

y x f
s
) , (

) , (
) , (
v u F
v u G
v u H
s
s
s
=

Department of Computer Science and Engineering, Hanyang University
Estimation by experimentation
Possible only if equipment similar to the equipment
used to acquire the degraded images is available

Eg. Use an impulse

Department of Computer Science and Engineering, Hanyang University
Estimation by modeling
Based on either physical characteristics or basic
principles

Eg.1. Physical characteristics: atmospheric turbulence


Eg.2. Math derivation: motion blur
Starting from


After some manipulation(p.259)


Setting the motion model, we obtain the degradation
func.

6 / 5
) (
2 2
) , (
v u k
e v u H
+
=
dt e v u H
T
t vy t ux j
}
+
=
0
)] ( ) ( [ 2
0 0
) , (
t
}
=
T
dt t y y t x x f y x g
0
0 0
)] ( ), ( [ ) , (

Department of Computer Science and Engineering, Hanyang University
Eg.1. Physical model
Atmospheric turbulence

Department of Computer Science and Engineering, Hanyang University
Eg.2. Math modeling
Motion blur

Department of Computer Science and Engineering, Hanyang University
Restoration methods
Inverse filtering
Wiener filtering
Constrained least square filtering
Geometric mean filtering
Etc..

Department of Computer Science and Engineering, Hanyang University
Inverse filtering




Poor performance!
Very sensitive to noise



) , ( ) , ( ) , ( ) , ( v u N v u F v u H v u G + =
) , (
) , (
) , ( ) , (

v u H
v u N
v u F v u F + =
) , (
) , (
) , (

v u H
v u G
v u F =
Noise amplification
when H(u,v) is small

Department of Computer Science and Engineering, Hanyang University
Eg. Inverse filtering

Department of Computer Science and Engineering, Hanyang University
Minimum mean-square error filter
Necessary to handle noise explicitly
Statistical characteristics of noise should be
incorporated into the restoration process
MMSE filter
To find an estimate of the uncorrupted image
such that the mean square error between them is
minimized:

Assume:
the noise and the image are uncorrelated
The one or the other has zero mean
The gray levels in the estimate are a linear function of
the levels in the degraded image
Derivation: Homework
} )

{(
2 2
f f E e =
f

f

Department of Computer Science and Engineering, Hanyang University
MMSE filter (cont.)
Frequency domain expression:







Approximation of the Wiener filter
) , (
) , ( / ) , ( | ) , ( |
) , ( *

) , (
) , ( | ) , ( | ) , (
) , ( ) , ( *
) , (

2
2
v u G
v u S v u S v u H
v u H
v u G
v u S v u H v u S
v u S v u H
v u F
f
f
f
(
(

+
=
(
(

+
=
q
q
Wiener filter
) , (
| ) , ( |
| ) , ( |
) , (
1
) , (

2
2
v u G
K v u H
v u H
v u H
v u F
(

+
=
PS of noise PS of image f

Department of Computer Science and Engineering, Hanyang University
Eg. Wiener filtering
Using the approximation
K is chosen interactively

Department of Computer Science and Engineering, Hanyang University
Eg. Restoration by Wiener filter
motion blur
Severe noise
Moderate noise
Negligible noise

Department of Computer Science and Engineering, Hanyang University
Constrained Least Square Filtering
Difficulty in Wiener filter
The power spectra of the undegraded image and noise
must be known
Minimization in a statistical sense


The constrained LS filtering
requires knowledge of
Mean of the noise
Variance of the noise
Optimal result for each image


Department of Computer Science and Engineering, Hanyang University
Vector-matrix form of convolution
g: MN-vector (lexicographical order of an image)
f: MN-vector
H: MNxMN matrix
Hf g + =

Department of Computer Science and Engineering, Hanyang University
Formulation: Constrained LS filter

To find the minimum of a criterion function C defined as



subject to the constraint


where is the Euclidean vector norm



| |

=
V =
1
0
1
0
2
2
) , (
M
x
N
y
y x f C
2 2
|| || ||

|| f H g =
w w w
T

2
|| ||

Department of Computer Science and Engineering, Hanyang University
Freq. Domain Sol.
: adjustable parameter
: Fourier transform of the Laplacian operator
) , (
| ) , ( | | ) , ( |
) , ( *
) , (

2 2
v u G
v u P v u H
v u H
v u F
(

+
=

) , ( v u P

Department of Computer Science and Engineering, Hanyang University
Eg. Constrained LS filter
Significant improvement over Wiener filter

Department of Computer Science and Engineering, Hanyang University
Procedure for computing
Define a residual vector


Adjust so that


Calculation

a = =
2 2
|| || || || ) ( r |
f H g r

=
] [ || ||
2 2 2
q q
o m MN + =
In general, automatically determined restoration filter
yields inferior results to manual adjustment of filter
parameters
iteration