This lecture: models of shape and texture

Active Shape Models

Eigenfaces
Active Appearance Models
Class Projects

Adjust to texture

Fit to shape model

1. Active Shape Models

Automatic Erosion Detection in RA Monitoring | Georg Langs

Example: bone contours

Task: localize anatomical structures

2 2

Task: Analyze a hand radiograph

Automatic Erosion Detection in RA Monitoring | Georg Langs

Task: Analyze a hand radiograph

Assume: we are
looking for proximal
phalanx 2
PP2

MC2

MP2

PP3

MC3
MC4

PP4

MP3
MP4

MC5
PP5

MP5

Automatic Erosion Detection in RA Monitoring | Georg Langs

Analyzing a hand radiograph

PP2

We have a priori knowledge

about the typical appearance:
e.g. bone shapes and texture

How can we represent this knowledge?

How can weAutomatic
exploitErosion
it? Detection in RA Monitoring | Georg Langs

How to capture a priori knowledge?

Simple shape representation: shape
consists of a set of points defined by
their individual coordinates
very powerful
very general
very unspecific
Alternative:
capture common properties of the
bone
find a representation that is restricted
to plausible bones.
Automatic Erosion Detection in RA Monitoring | Georg Langs

Prior knowledge: first works

Snakes

M. Kass, A. Witkin and

D. Terzopoulos,
`Snakes, ICCV 1987,
IJCV 88

Data-driven
Prior knowledge

Deformable templates

A.L.Yuille, D. Cohen, and P.Hallinan,

Feature extraction from faces using
deformable templates, CVPR 89,
IJCV 92.

Current work: Statistical Shape Models

Each example is represented by a

vector containing the coordinates
of the landmarks.

Learning: Model Acquisition

Inference: Model Fitting

1. Active Shape Models

Model learning
How can we learn what a bone looks like?
To build a statistical model of the data
we need many examples.

Objectives:

Expressiveness (model can account for all bones)

Compactness (as simple as possible)

Method: Principal Components Analysis (PCA)

9

The space of all bone shapes

Bone shapes: vectors in

Goal: project data onto a low-dimensional linear

subspace that best explains their variation.

PCA: feature space

1. Active Shape Models

11

PCA: probability distribution

Example with
high probability

Example with
low probability

1. Active Shape Models

12

New subspace: `better coordinate system

New coordinates reflect

the distribution of the
data.
Mean

Few coordinates suffice to

represent a high
dimensional vector
They can be viewed as
parameters of a model
1. Active Shape Models

13

Principal Component Analysis

Given: N data points x1, ,xN in Rd
We want to find a new set of features that are linear
combinations of original ones:
u(xi) = uT(xi )
(: mean of data points)

What unit vector u in Rd captures the most variance of

the data?

Principal Component Analysis

Variance of projection on u:
N

Projection of data point

N

Covariance matrix of data

Direction: Unit norm vector

The direction that maximizes the variance: the eigenvector associated
with the largest eigenvalue of

Principal component analysis

The direction that captures the maximum covariance of

the data is the eigenvector corresponding to the largest

eigenvalue of the data covariance matrix

Furthermore, the top k orthogonal directions that

capture the most variance of the data are the k
eigenvectors corresponding to the k largest eigenvalues

Covariance matrix reminder

Covariance matrix:

Uncorrelated coordinates: diagonal covariance

Height, Income

Height, Weight

PCA: decorrelation of random variables

PCA: projection onto eigenvectors of covariance matrix

Dimensionality
reduction by using
only leading
eigenvectors

18

PCA: step by step

1. Compute the empirical mean and subtract it from the
data, and compute empirical covariance matrix
2. Compute eigenvalue decomposition of the covariance
matrix resulting in

3. Retain only the k eigenvectors with highest

corresponding eigenvalues.

1. Active Shape Models

PCA: step by step

4. Given a new input vector

5. project it into the eigenspace

Only first k
eigenvectors

6. resulting in the coefficient vector or parameter vector

1. Active Shape Models

Using PCA to model shape

1. Active Shape Models

++

21

Active shape models (ASM)

A set of training examples (images)
A set of landmarks, that are present on
all images
Build a statistical model of shape
variation (PCA)
Build a statistical model of the local
texture (PCA)
Use the model for the search in a new
image

1. Active Shape Models

22

ASM: model building

Training set:
Instances vary in
- shape
- texture
- orientation
pose
- scale
- position

m Landmarks
represented as

1. Active Shape Models

23

ASM: alignment of examples

s2

s1
center of
gravity

scale

center
(translate)

rotate

The alignment removes rotation, scale and translation differences

Minimizes Procrustes distance:

1. Active Shape Models

24

ASM: alignment of examples

Center all training shapes
Choose one example as initial mean shape
and scale

Align all shapes

to
by minimizing the
Procrustes distance
Recalculate the mean shape
from the aligned
set.
Iterate until the mean shape converges

1. Active Shape Models

25

ASM: shape model

Given: a set of aligned shapes
Perform PCA on the set of shapes, resulting in
a set of Eigenvectors and corresponding
Eigenvalues

Each shape can be reconstructed by

1. Active Shape Models

26

ASM search

Initialize
Adjust to texture
1. Active Shape Models

Fit to shape model

27

ASM search: shape

Update the instance according to shape model and pose
Initialize

Generate model pose instance

Find pose parameters to map
to
Project into the model coordinates
Update model parameters to match
Apply constraints

1. Active Shape Models

28

ASM search

1. Active Shape Models

29 29

Modelling the appearance

To peform a search with the shape model, we have to
capture the appearance, too.

Local appearance at the landmark positions:

Active shape model (ASM)

Appearance of the enclosed texture

Active appearance model (AAM)

Automatic Erosion Detection in RA Monitoring | Georg Langs

30

This lecture: models of shape and texture

Active Shape Models

Eigenfaces
Active Appearance Models
Class Projects

=
^
x

w1u1+w2u2+w3u3+w4u4+

Automatic Erosion Detection in RA Monitoring | Georg Langs

31

Face Recognition Problem

Face Recognition Problem

[Sinha and Poggio 2002]

Challenges: Image Variability

Short Term

Expression

Pose

Illumination

Long Term

Facial Hair
Makeup
Eyewear

Hairstyle
Piercings
Aging

Invariance Problem

Invariance Problem

Images as Points in Euclidean Space

x

Let an n-pixel image to be a point in an n-D space, x ! Rn.

Each pixel value is a coordinate of x.

Face Recognition: Euclidean Distances

D1 > 0

D2 > 0

D3 ~
=0

Face Recognition: Euclidean Distances

D1 > 0

D2 > 0

D3 > D1 or 2
[Hallinan 1994] [Adini, Moses, and Ullman 1994]

Nearest Neighbor Classifier

Variations on distance function (e.g. L1, robust

distances)
Multiple templates per class- perhaps many
training images per class.
Expensive to compute k distances, especially
when each image is big (N dimensional).
May not generalize well to unseen examples.
Some solutions:

Bayesian classification
Dimensionality reduction

Image Variability: Appearance Manifolds

x2

xn
x1

Lighting x Pose

[Murase and Nayar 1993]

Image Variability: From Few to Many

Lighting x Pose

x2

xn
x1

Real
Synthetic
[Georghiades, Belhumeur, and Kriegman 1999]

Modeling Image Variability

Can we model illumination and pose variability in
images of a face?

Yes, if we can determine the shape and texture of

the face. But how?

Appearance modelling for faces

When viewed as vectors of pixel values, face images are
extremely high-dimensional

100x100 image = 10,000 dimensions

Very few vectors correspond to valid face images

Original coordinates are not revealing about face propertie

We want to model the subspace of face images
Same story as shape modelling.

Eigenfaces: Key idea

Assume that most face images lie on

a low-dimensional subspace determined by the first k

(k<d) directions of maximum variance

Use PCA to determine the vectors or eigenfaces u1

,uk that span that subspace

Represent all face images in the dataset as linear

combinations of eigenfaces

Same idea as shapes

M. Turk and A. Pentland, Face Recognition using Eigenfaces, CVPR 1991

Eigenfaces example
Training images
x1,,xN

Eigenfaces example
Top eigenvectors: u1,uk

Mean:

Eigenfaces example
Principal component (eigenvector) uk

+ 3kuk

3kuk

Eigenfaces example
Face x in face space coordinates:

Eigenfaces example
Face x in face space coordinates:

=
Reconstruction:
=
^
x

w1u1+w2u2+w3u3+w4u4+

Recognition with eigenfaces

Process labeled training images:

Find mean and covariance matrix

Find k principal components (eigenvectors of ) u1,uk
Project each training image xi onto subspace spanned by
principal components:
(wi1,,wik) = (u1T(xi ), , ukT(xi ))

Given novel image x:

Project onto subspace:

(w1,,wk) = (u1T(x ), , ukT(x ))
Check reconstruction error x ^x to determine whether image
is really a face
Classify as closest training face in k-dimensional subspace

M. Turk and A. Pentland, Face Recognition using Eigenfaces, CVPR 1991

Challenges addressed by Eigenfaces

Short Term

Expression

Pose

Illumination

Long Term

Facial Hair
Makeup
Eyewear

Hairstyle
Piercings
Aging

Limitations
Global appearance method: not robust to misalignment,
background variation

Linear Discriminant Analysis

We wish to choose linear functions of the features that
allow good discrimination.

Assume class-conditional covariances are the same

Want linear feature that maximises the spread of class means for
a fixed within-class variance

This lecture: models of shape and texture

Active Shape Models

Eigenfaces
Active Appearance Models
Class Projects

Automatic Erosion Detection in RA Monitoring | Georg Langs

61

Challenges: Image Variability

Short Term

Expression

Pose

Illumination

Long Term

Facial Hair
Makeup
Eyewear

Hairstyle
Piercings
Aging

Modelling Shape Variability via Deformations

DArcy Thompson, On Growth and Form (1917)

Modelling Shape Variability via Deformations

DArcy Thompson, On Growth and Form (1917)

Active Appearance Models

Appearance = Shape + Texture
Shape: deformation from `template domain to
instance domain

I(S (x)) = T (x)

X

T emplate

S (X )

Ins tance

Texture: intensity (or color) patch of an image in the

template domain.

Shape-free texture
An attempt to eliminate texture variation

due to shape
Given instance I and a reference template
T we warp our image so that points of I
move into the corresponding points of T
I

warped to

becomes

AAM training
Training set: annotated grayscale images
Shape: displacements

Texture (shape-free)

from mean shape

PCA

Optional:
Interpolate to recover
dense deformation field

Model of shape

PCA

Model of texture

Active Appearance Models

Shape:

Appearance:
Synthesis:
I(S (x)) = T (x)
X

T emplate

S (X )

Ins tance

AAM model
Shape Eigenvectors

Texture Eigenvectors

Playing with the Parameters

First two modes of shape variation

First two modes of gray-level variation

First four
modes of
appearance
variation

Active Appearance Model fitting

Given:
1) an appearance model,
2) a new image,
3) a starting approximation

Find:
the best matching
synthetic image

Minimize reconstruction error

How?

Alternate between estimating s and t

For t: projection of deformed image

onto PCA basis
For s?

AAM parameter estimation: shape

Iterative scheme (Newton-Raphson minimization)

72

Active Appearance Model Search (Results)

Model fitting + Image segmentation

AAM fitting

Variational Segmentation

Model fitting + Image segmentation

EM formulation

E-step: Assign data to models

M-step: Estimate model parameters

Joint segmentation & recognition

E-step: segmentation
M-step: deformable model fitting
Atomic
Regions

M-step

E-step

Challenges: Image Variability

Short Term

Expression

Pose

Illumination

Long Term

Facial Hair
Makeup
Eyewear

Hairstyle
Piercings
Aging

Modeling Image Variability: 3-D Faces

Laser Range Scanners

Stereo Cameras
Structured Light
Photometric Stereo

What can we do with 3d shape?

[Blanz and Vetter 1999, 2003]

Building a Morphable Face Model

[Blanz and Vetter 1999, 2003]

3-D Morphable Models

[Blanz and Vetter 1999, 2003]

3D Morphable models

Recover Shape

Synthesize new views

Synthesize new expressions

3-D Morphable Models fitting

Rough manual initialization

Gradient descent to minimize reconstruction
error functional

And then

82

3D AAM for face tracking

CMU group: I. Matthews, S. Baker, R. Gross

(230 Frames per second)

83

3D AAM for face tracking

84

Playing with Facial Attributes

Several classes of attributes are modeled:
Facial expressions (smile, frown)
Individual characteristics (double chin, hooked
nose, maleness)
Distinctiveness

Manipulating Facial Attributes via Deformations

For each face in the database, two scans are recorded:

Sneutral, and Sexpression.

The difference vector DS = Sexpression - Sneutral is saved and
later on simply added to the 3D reconstruction of the input
image.

88