Feature Extraction and Face

Recognition
Dr. Rama Krishna Sai S Gorthi
Image Features
 How do we determine a physical point
uniquely in different images
 How to find the transformation between two
images, that are take from different
positions
 Using features which favor the decision
making process.
 Feature can be anything like color, size,
aspect ratio, orientation, …
`
Feature Extraction
 Well known feature extractors
 Moravac operator
 Harries corner detector
 Scale Invariant Feature Transform (SIFT)
 Properties of a good feature extractor
 Scale, rotation, illumination invariant
 Robust to noise
 Robust to transformations (such as affine..)
Response of Moravec operator
Moravec Operator
Output of Moravec Operator
Harris/Plessey corner detector
Input: gray scale image, Gaussian variance
(window typically has a radius of 3 times the
standard deviation), k value, threshold T
 1. For each pixel (x, y) in the image
calculate the autocorrelation matrix M:
Harris/Plessey corner detector
2. Construct the cornerness map by calculating the cornerness
measure C(x, y) for each pixel (x, y):

3. Threshold the interest map by setting all C(x, y) below a

threshold T to zero.
4. Perform non-maximal suppression to find local maxima.

All non-zero points remaining in cornerness map are corners.

Harris results
Harris results
Anisotropic response of Harris
detector
 Interest Point Descriptor
what features should we select for the interest points detected?
Interest Point Descriptor

 The descriptor need to be unique.

 The descriptor should be robust to noise.

 The computation of the descriptor should be computationally feasible.

 The descriptor need to be invariant to

 Scale
 Rotation
 Illumination
 Brightness …
SIFT/ HoG Interest Point
Descriptor
Image features in classification
Image Registration / Stitching /
Mosaicking – Feature matching
Image/Video Masaicing
 Consider Consecutive frames of a video
 Estimate the global motion between the frames
 using feature point correspondences or

 Approximate with Rigid body transformation or

Affine or Projective
 Consider first frame as reference
 Warp the second frame with estimated motion
SIFT Interest Point Descriptor
Image Mosaicing

 If we could find at least 3 good correspondences the affine relationship

between the images can be estimated and can be used for image
alignment.
Image Mosaicking : Affine
Face Detection – Feature based
Ensemble classification
Viloa-Jones Face Detection Algorithm
Face Detection Steps:
1. Haar Features
2. Integral image
3. Adaboost algorithm (Ensemble classifier)
4. Cascading: Most of the patches in image are Face less and can
be quickly or easily discarded. Attention is paid proportion to
information or related feature content.
Stage 1 Stage 2
May be May be
Is input a face? Is input a face?

Definetily not Definetily not

Discard input Discard input

HAAR FEATURES:

Human faces share some similar properties and it used to

construct certain features known as Haar Features.

The properties that are similar for a human face are:

1. The eyes region is darker than the upper-cheeks.
2. The nose bridge region is brighter than the eyes.
3. Location - Size: eyes & nose bridge region
4. Value: darker / brighter
1. Haar features:
 Value = ∑ (pixels in
white area) - ∑ (pixels in
black area)
 Three types: two-, three-,
four-rectangles
 For example: the
difference in brightness
between the white and
black rectangles over a
specific area
2. Each feature is related to a
special location in the sub-
window. Each feature may
have any size
3. If we consider the Haar features like position, scale and type
we have to calculate up to 160,000+ features.

Type 1 Type 2 Type 3 Type 4 Type 5

Haar features used in Viola Jones method applying on a

given image
 Integral image
 Integral image technique is used for faster feature evaluation. In
an integral image the value of value at (x,y) is sum of the pixels
above and left to of (x,y).
Formal definition:
ii 𝑥, 𝑦 = 𝑥 ′≤𝑥,𝑦′≤𝑦 𝑖(𝑥 ′ , 𝑦′)
Recursive definition:
s(x,y) = s(x,y-1)+i(x,y)
ii(x,y) = ii(x-1, y)+ s(x,y)

S(x,y) is the cumulative row sum, i(x,y) is the original image

Image
Integral image
0 1 1 1 0 1 2 13
1 2 2 3 1 4 7 11
1 2 1 1 2 7 11 16
1 3 1 0 3 11 16 21
Integral image allows for the calculation of sum of
the pixels inside any given rectangle using only four
values at the corner of the rectangle

Sum of all pixels in

D = 1+4-(2+3)
= A+(A+B+C+D)-(A+C+A+B)
=D
 Starting with the feature being a 1 x 2 in size and in position
x1,y1, then we have the first feature
 Then move on to the next size, which would be a 1 x 4 in
position x1, y1, be the second feature
 By increasing in size it would be 1 x 8, 1 x 16 and so on.
 Then the classifiers would start with a 2 x 2 and it would
continue 2 x 6, 2 x 8, and so on.
 So for 24×24 image has 160,000+ features.
Ada-boost algorithm: Feature Selection

 Adaboost is algorithm which helps in finding only

the best features among all the 160,000+ features.
 After these features are found, a weighted
combination of all features is used in evaluating
and deciding any given window has a face or not.
 Adaboost finds the single rectangular feature and
threshold that separates the faces and non faces
training samples
Face Detection Results
Face Detection Results
Measuring Optical flow
Optical Flow Estimation

 How to measure movement of a world point

(pixel) from frame to frame : (u,v)
 use photometric invariance or Brightness
consistency equation
 Aperture Problem
 Horn-Shunk optical flow
 Incorporates a regularization term
 Lucas-Kanade optical flow
 Assumes locally constant optical flow
Optical Flow: Horn and Schunk
 Tracking objects
Found a suspect. How will I track him?
Object Tracking

 Estimate the motion (optical flow) at points of interest (points

that defines the object).

 Connect the flow vectors in subsequent frame to get the track of

the object.
 Feature Matching
How will I compare different feature vectors?
Stereo analysis
Depth estimation from image pair
Conclusions
 Computer vision techniques can be
developed to various real world tasks
 Few more prominent computer vision
tasks/applications include
 Object, Face and Gusture recognition
 Human detection
 Tracking people, cars etc.
 3D Reconstruction
References
 Video lectures of Dr. Mubarak Shaw
 Lecture slides of Dr. Mubarak Shaw
 Video lectures of Dr. Jitendra Mallick
 Computer vision principles, algorithms, and applications by Richard
Sezelski
 Viola, Paul, and Michael J. Jones. "Robust real-time face detection."
International journal of computer vision 57.2 (2004): 137-154.
 Viola, Paul, and Michael Jones. "Robust real-time object detection."
International Journal of Computer Vision 4 (2001): 34-47.
Thank you !