Video Modeling

(Basics: Camera Models and Motion Models)

Two-Dimensional Motion Estimation
(Fundamentals & Basic Techniques)
Chapter 5 (parts) and Chapter 6
2
Outline (chapter 5)
Camera projection
3D motion
Projection of 3-D motion
2D motion due to rigid object motion
Projective mapping
Approximation of projective mapping
Affine model
Bilinear model

3
Pinhole Camera
Camera
center
Image
plane
2-D
image
3-D
point
The image of an object is reversed from its
3-D position. The object appears smaller
when it is farther away.
4
Pinhole Camera Model:
Perspective Projection
Z y x
Z
Y
F y
Z
X
F x
Z
Y
F
y
Z
X
F
x
to related inversely are ,
, , = = = =
All points in this ray will
have the same image
5
Approximate Model:
Orthographic Projection
images) c microscopi i.e. Z, F for is Y y and X (x
object. the of distance the to compared
small is object in the ation with depth vari the as long as used be Can
const when F/Z with ic orthograph scaled sY, y sX, actually x
, ,
) ( far very is object When the
>> = =
A >> = = =
= =

Z Z
Y y X x
Z
6
Rigid Object Motion
z y x z y x
T T T , , : ; , , : ] [ ; ) ]( [ '
: center object the on wrp. translati and Rotation
T R C T C X R X u u u + + =
7
Rigid Object Motion
] , , [
z y x
T T T = T
z y x z y x
T T T , , : ; , , : ] [ ; ) ]( [ '
: center object the on wrp. translati and Rotation
T R C T C X R X u u u + + =
(
(
(

+
(
(
(

(
(
(

=
(
(
(

z
y
x
T
T
T
Z
Y
X
r r r
r r r
r r r
Z
Y
X
9 8 7
6 5 4
3 2 1
'
'
'
8
Flexible Object Motion
Two ways to describe
Decompose into multiple, but connected rigid sub-objects
Global motion plus local motion in sub-objects
Ex. Human body consists of many parts each undergo a rigid
motion
Elastic motion cant easily decomposed
Mesh models
9
3-D Motion -> 2-D Motion
2-D MV
3-D MV
Sample (2D) Motion Field
11
Occlusion Effect
Motion is undefined in occluded regions
12
Typical Camera Motions
13
2-D Motion Corresponding to
Camera Motion
Camera zoom
Camera rotation around Z-axis (roll)
14
2-D Motion Corresponding to
Rigid Object Motion
General case:






Projective mapping:
(if not planar planar patches)

F T Z F r y r x r
F T Z F r y r x r
F y
F T Z F r y r x r
F T Z F r y r x r
F x
T
T
T
Z
Y
X
r r r
r r r
r r r
Z
Y
X
z
y
z
x
z
y
x
+ + +
+ + +
=
+ + +
+ + +
=

(
(
(

+
(
(
(

(
(
(

=
(
(
(

) (
) (
'
) (
) (
'
'
'
'
9 8 7
6 5 4
9 8 7
3 2 1
Proj ection e Perspectiv
9 8 7
6 5 4
3 2 1
y c x c
y b x b b
y
y c x c
y a x a a
x
2 1
2 1 0
2 1
2 1 0
1
' ,
1
'
: c) bY aX (Z object When the planar is surface
+ +
+ +
=
+ +
+ +
=
+ + =
15
Projective Mapping
Two features of projective mapping:
Chirping: increasing perceived spatial frequency for far away objects
Converging (Keystone): parallel lines converge in distance
16
Affine and Bilinear Model
Approximations to Projective Mapping
Affine (6 parameters):


Good for mapping triangles to triangles

Bilinear (8 parameters):

Good for mapping blocks to quadrangles
Polynomial models (biquadratic 12 parameters)
(

+ +
+ +
=
(

y b x b b
y a x a a
y x d
y x d
y
x
2 1 0
2 1 0
) , (
) , (
(

+ + +
+ + +
=
(

xy b y b x b b
xy a y a x a a
y x d
y x d
y
x
3 2 1 0
3 2 1 0
) , (
) , (
17
Motion Field Corresponding to
Different 2-D Motion Models
Translation
Bilinear
Perspective
Affine
Outline (Chapter 6)
Optical flow equation and ambiguity in motion estimation
General methodologies in motion estimation
Motion representation
Motion estimation criterion
Optimization methods
Gradient descent methods
Pixel-based motion estimation
Block-based motion estimation
EBMA algorithm
19
2-D Motion vs. Optical Flow
On the left, a sphere is rotating
under a constant ambient
illumination, but the observed image
does not change.
On the right, a point light source is
rotating around a stationary
sphere, causing the highlight point
on the sphere to rotate.
2-D Motion: Projection of 3-D motion, depending on 3D object motion and
projection operator
Optical flow: Perceived 2-D motion based on changes in image pattern,
also depends on illumination and object surface texture
20
Optical Flow Equation
When illumination condition is unknown, the best one
can do it to estimate optical flow.
Constant intensity assumption -> Optical flow equation

0 or 0 or 0
: equation flow optical the have we two, above the Compare
) , , ( ) , , (
: expansion s Taylor' using But,
) , , ( ) , , (
: " assumption intensity constant " Under
=
c
c
+ V =
c
c
+
c
c
+
c
c
=
c
c
+
c
c
+
c
c
c
c
+
c
c
+
c
c
+ = + + +
= + + +
t t
v
y
v
x
d
t
d
y
d
x
d
y
d
y
d
x
t y x d t d y d x
t y x d t d y d x
T
y x t y x
t y x t y x
t y x

v
21
Optical Flow Equation

) , , ( of ector gradient v spatial the is ] , [
vector velocity the is ) v , ( where
0 or 0
equation flow optic is That
y
t y x
y x
v
t t
v
y
v
x
T T
x
T
y x

c
c
c
c
= V
=
c
c
+ V =
c
c
+
c
c
+
c
c
v
22
Ambiguities in Motion Estimation
Optical flow equation only
constrains the flow vector
in the gradient direction
The flow vector in the
tangent direction ( ) is
under-determined
In regions with constant
brightness ( ), the
flow is indeterminate ->
Motion estimation is
unreliable in regions with
flat texture, more reliable
near edges


n
v
0 = V
0 =
c
c
+ V
+ =
t
v
v v
n
t t n n

e e v
t
v
23
Ambiguities in Motion Estimation
n
v
0 = V
t
v

The flow vector in the tangent direction ( Vt ) is
under-determined (aperture problem)

24
Ambiguities in Motion Estimation

The flow vector in the tangent direction ( Vt ) is
under-determined (aperture problem)

25
General Considerations
for Motion Estimation
Two categories of approaches:
Feature based (more often used in object tracking, 3D
reconstruction from 2D)
Intensity based (based on constant intensity assumption)
(more often used for motion compensated prediction,
required in video coding, frame interpolation) -> Our focus
Three important questions
How to represent the motion field?
What criteria to use to estimate motion parameters?
How to search motion parameters?
26
Motion Representation
Global:
Entire motion field is
represented by a few
global parameters
Pixel-based:
One MV at each pixel,
with some smoothness
constraint between
adjacent MVs.
Region-based:
Entire frame is divided
into regions, each
region corresponding
to an object or sub-
object with consistent
motion, represented by
a few parameters.
Block-based:
Entire frame is divided
into blocks, and motion
in each block is
characterized by a few
parameters.

Also mesh-based
(flow of corners,
approximated inside)
27
Notations
Anchor frame:
Target frame:
Motion parameters:
Motion vector at a
pixel in the anchor
frame:
Motion field:
Mapping function:
) (
1
x
) (
2
x
) (x d
A e x a x d ), ; (
a
A e + = x a x d x a x w ), ; ( ) ; (
T
L
a a a ] ,..., , [
2 1
= a
Backward motion estimation
Forward motion estimation
28
Motion Estimation Criterion
To minimize the displaced frame difference (DFD)
(MSE) Error Square Mean , min ) ( )) ; ( ( ) (
(MAD) Difference Absolute Mean , min ) ( )) ; ( ( ) (
2
1 2
1 2
MSE - DFD
MAD - DFD
+ =
+ =

A e
A e
x
x
E
E
x a x d x a
x a x d x a


( )
( ) min ) ( ) ( ) ; ( ) ( ) (
0 ) (
) ( ) ( small is if 0
2
1 2 1 OF
1 2 1
1 2
1 1 1
+ V =
= + V
~
c
c
=
c
c
+
c
c
+
c
c

A e
=
x
p
T
T
t t t y x
E
d
t
d d
t
d
y
d
x
x x a x d x a
d
x x




To satisfy the optical flow equation

29
Motion Estimation Criterion
( )
1
0 1 1 1 2 1
1
2
2
( ( ( )) ( ) ) ( ( ( ) ( )) ( ))
( ) ( )( ) ( )( )
( )( ) ( )( ) ( )
T
x x
x x x x
y
x x x
v
x x y x t
v
y t x y y

o o o o o
o o o o o
o o o o o
o o o o o

eA eA

eA eA eA
eA eA eA
= V V V
( (

( (
(
( (
=
(
( (


( (




d x x x x x
Block size: 8*8 16*16
This is homework
30
Motion Estimation Criterion
Two above methods fail
Constant intensity assumption is not true everywhere (say shadows)
Flat texture: many motions satisfy the assumption (ill-posed)
Need to impose additional smoothness constraint using regularization
technique (Important in pixel- and block-based representation)
Penalty term for smoothness, measure motion flow differences between
adjacent pixels
use weights to include a priori knowledge of importance of smoothness
min ) ( ) (
) ; ( ) ; ( ) (
DFD
2
+
=

A e e
a a
a y d a x d a
x y
s s DFD
N
s
E w E w
E
x
31
Relation Among Different Criteria
OF criterion is good only if motion is small.
OF criterion can often yield closed-form solution as the objective
function is quadratic in MVs.
When the motion is not small, can iterate the solution based on
the OF criterion to satisfy the DFD criterion.
Bayesian criterion can be reduced to the DFD criterion plus
motion smoothness constraint
More in the textbook
32
Optimization Methods
Exhaustive search
Typically used for the DFD criterion with p=1 (MAD)
Guarantees reaching the global optimal
Computation required may be unacceptable when number of
parameters to search simultaneously is large!
Fast search algorithms reach sub-optimal solution in shorter time
Gradient-based search
Typically used for the DFD or OF criterion with p=2 (MSE)
the gradient can often be calculated analytically
When used with the OF criterion, closed-form solution may be obtained
Reaches the local optimal point closest to the initial solution
Multi-resolution search
Search from coarse to fine resolution, faster than exhaustive search
Avoid being trapped into a local minimum
33
Gradient Descent Method
Iteratively update the current estimate in the direction opposite the gradient
direction.












The solution depends on the initial condition. Reaches the local minimum closest
to the initial condition
Choice of step side:
Fixed step size: Step size must be small to avoid oscillation, requires many iterations
Steepest gradient descent (adjust step size optimally)
Not a good initial
A good initial
Appropriate
stepsize
Stepsize
too big
34
Newtons Method
Newtons method








Converges faster than 1
st
order method (I.e. requires fewer number of iterations to
reach convergence)
Requires more calculation in each iteration
More prone to noise (gradient calculation is subject to noise, more so with 2
nd
order
than with 1
st
order)
May not converge if \alpha >=1. Should choose \alpha appropriate to reach a good
compromise between guaranteeing convergence and the convergence rate.

35
Newton-Raphson Method
Newton-Ralphson method
Approximate 2
nd
order gradient with product of 1
st
order gradients
Applicable when the objective function is a sum of squared errors
Only needs to calculate 1
st
order gradients, yet converge at a rate similar to
Newtons method.