BSC Thesis Todor Balabanov NBU 2006

NEW BULGARIAN UNIVERSITY
BACHELOR PROGRAM INFORMATICS

Bachelor Thesis
Library for image co-registration with

application for mosaicing
by
Todor Dimitrov Balabanov
Faculty No: F19952
Promoter:
Prof. Hichem Sahli
hsahli@etro.vub.ac.be
Supervisors:
Dr. Frank Cremer
fcremer@etro.vub.ac.be
Dr. Lixin Yang
lxyang@etro.vub.ac.be
Reviewer:
Assoc. Prof. Stanislav Ivanov
sivanov@nbu.bg
September, 2006
Sofia
Table of contents
Library for image co-registration with application for mosaicing..............................................1
Table of contents......................................................................................................................... 2
1
Introduction.........................................................................................................................3
1.1
Problem Description................................................................................................... 3
1.2
Structure of the report................................................................................................. 4
2
Library overview.................................................................................................................5
2.1
Structure...................................................................................................................... 5
2.2
Calling conventions.................................................................................................. 10
input/output variable................................................................................................. 11
2.3
Data types/structure.................................................................................................. 12
2.4
Functions list............................................................................................................. 12
2.5
Memory allocations.................................................................................................. 17
3
Optical flow based registration......................................................................................... 18
3.1
Theory....................................................................................................................... 18
Dense optic flow....................................................................................................... 19

3.2
Implementation......................................................................................................... 20
3.3
Experiments and discussion...................................................................................... 20
4
Pixel migration based registration.................................................................................... 23
4.1
Theory....................................................................................................................... 23
Iterative optimization................................................................................................ 24
Genetic Algorithms................................................................................................... 25
Multiscale extension................................................................................................. 27
4.2
Implementation......................................................................................................... 27
4.3
5
Mutual information based registration..............................................................................38
5.1
Theory....................................................................................................................... 38
5.2
Nonlinear conjugate gradient method....................................................................... 40
5.3
Implementation......................................................................................................... 41
5.4
6
User interface.................................................................................................................... 44
6.1
Overview...................................................................................................................44
6.2
Tutorial......................................................................................................................44
Input of images......................................................................................................... 45
Selection of registration methods and parameters.................................................... 46
Visualization of results..............................................................................................48
Limitations of user interface..................................................................................... 49

7
Mosaicing..........................................................................................................................50
7.1
Background............................................................................................................... 50
Why we need mosaic construction in ClearFast project?......................................51

7.2
Theory and formulations...........................................................................................51
7.2.1
Ground projection............................................................................................. 52
7.2.2
Histogram equalization..................................................................................... 54
7.2.3
Mosaic update................................................................................................... 56
2
7.2.4
Algorithm.......................................................................................................... 58
7.3
8
Conclusions.......................................................................................................................64
References.................................................................................................................................65
1 Introduction
1.1 Problem Description
The goal of this work is to provide C library with three different image co-registration
methods: Optic flow, Mutual information and Pixel migration. User interface in Matlab is
implemented for general testing and performance evaluation. The library provides also
routines for mosaic construction. Our work is based on some existing C and Matlab code [4]
[36][50].
Image registration is the process of spatially aligning two images of a scene. We have a
reference image, which is the image that is kept unchanged, and an observation image, which
is the image that is transformed to spatially align with the reference image. A transformation
function is used to warp the observation image to take the geometry of the reference image.
Registration of images has been in practice since long before the age of digital computers.
Shortly after the photographic process was developed in 1839, the use of photographs was
demonstrated on topographical mapping [48]. Images acquired from hill-tops or balloons
were manually pieced together. After the development of airplane technology (1903)
aerophotography became an exciting new field. The limited flying heights of the early
airplanes and the need for large photo-maps, forced imaging experts to construct mosaic
images from overlapping photographs. This was initially done by manually mosaicing images
which were acquired by calibrated equipment. [25] The need for mosaicing continued to
increase later in history as satellites started sending pictures back to earth. Improvements in
computer technology became a natural motivation to develop computational techniques and to
solve related problems.
Registration of two images proceeds in the following steps: 1) preprocessing (image
smoothing, deblurring, edge sharpening, image segmentation, edge detection); 2) feature
selection (extracting points, lines, regions, templates, etc. from an image); 3) feature
correspondence (determining the correspondence between features in two images); 4)
determining the transformation function (from the coordinates of corresponding points
determining the transformation function); and 5) resampling (resampling the observation
image to the coordinate system of the reference image using the transformation function).
Image registration has an application in the following areas: Change detection; Image fusion;
Target recognition; Target localization; Depth perception; Image mosaicing; and Motion
estimation. In this work we will focus on image registration for mosaic construction.
Image mosaic construction is about stitching together adjacent images of a scene into an
image that displays a wider field of view. Image mosaicing can be used in a wide variety of
applications such as remote sensing, visual surveillance and virtual reality.
The construction of mosaic images and the use of such images on several computer
vision/graphics applications have been active areas of research in recent years. There have
been a variety of new additions to the classic applications mentioned above that primarily aim
to enhance image resolution and field of view. In computer graphics applications (e.g. [12])
images of the real world have been traditionally used as environment maps. In early
applications such environment maps were single images captured by fish-eye lenses or a
sequence of images captured by wide-angle rectilinear lenses used as faces of a cube [12].
Mosaicing images on smooth surfaces (e.g. cylindirical [6][31][44] or spherical [7][42][49])
allows an unlimited resolution also avoiding discontinuities that can result from images that
are acquired separately. Such immersive environments (with or without synthetic objects)
provide the users an improved sense of presence in a virtual scene. A combination of such
scenes used as nodes [6][7] allows the users to navigate through a remote environment.
Computer vision methods can be used to generate intermediate views [5][31] between the
nodes. As a reverse problem the 3D stucture of scenes can be reconstructed from multiple
nodes [7][22][40][43][51].
Both image co-registration and mosaic construction are complex problems, which involve a
lot of calculations. First we developed models in Matlab, the existing codes we reimplemented into C codes. Implementation was done carefully in a structured way to provide
stable source codes to be properly integrated in future projects. We also want to provide
opportunity of using the codes in Matlab. The large Matlabs extension capabilities allow us
to integrate easily our C codes. In the same time we used Matlab to develop proper graphic
user interface for testing and evaluation, which is another task in the project.
1.2 Structure of the report

This paper is organized as follows. In Section 2 we give the library overview, including
structure, calling conventions and memory allocation. Section 3 presents the optical flow
based registration method, including theory background, implementation and experimental
results. Next in Section 4 we introduce the pixel migration based registration method,
including theoretical background, implementation and some experimental results. Section 5 is
about the mutual information based registration method, presenting nonlinear conjugate
gradient method as an optimization approach. Section 6 presents the graphic user interface,
developed in Matlab for testing and analyzing co-registration results. In Section 7 we present
mosaic construction technique, including theory, implementation and experiments. We end the
paper with Section 8 including general conclusions and proposal for future work
2 Library overview
The goal of this work is to buildup a C/C++ library with image co-registration and mosaicing
functions and corresponding tools for visualization and evaluation. The library has an
interface to Matlab, so that its functions can be directly used there.
The library contains three different methods for image co-registration:
Optical flow based image registration [36].
Mutual information based image registration [50].
Pixel migration based image registration [24].
The library has mosaic construction functionality which is related to image registration
techniques:
Generation of mosaics.
2.1 Structure
The library is organized in five modules, three each co-registration method, one for the
mosaic construction and one for the user interface. All modules are organized in separate
directory, which contains the C code files and header files.
Library
imreg_GUI
imreg_OF
imreg_MI
imreg_PM
mosaicing
Figure 2.1 Image co-registration library structure.
Graphic user interface consists of four sub-modules. Each sub-module consists of .fig and .m
file.
imreg_GUI
imreg_UI
options_m
options_of
options_pm
Figure 2.2 Graphic user interface code structure.
imreg_OF
include
affine.h
bib_ansi.h
bib_LP_ansi.h
constantes.h
neighbor.h
of.h
relax_V3.h
types_4v_V1.h
src
affine.c
bib_ansi.c
bib_LP_ansi.c
imreg_OF.c
neighbor.c
of.c
relax_V3.c
trans_OF.c
NeedleMap.m
QuiverPrune.m
Figure 2.3 Optic flow code structure.
imreg_PM
include
errors.h
mxinv.h
pm.h
pm_ga.h
src
imreg_PM.c
mxinv.c
pm.c
pm_ga.c
gaussian.m
pixel_migration.m
pm_build_in_ga.m
pm_ga.m
pm_ncg.m
pm_resault.m
Figure 2.4 Pixel migration code structure.
mosaicing
include
errors.h
findparam.h
fov.h
fovpoint.h
genmosaic3a.h
histspec2.h
imtransform.h
loadsettings.h
mxinv.h
mxmul.h
projective.h
rad.h
vecnorm.h
src
findparam.c
fov.c
fovpoint.c
genmosaic3a.c
histspec2.c
imtransform.c
loadsettings.c
mxinv.c
mxmul.c
projective.c
rad.c
vecnorm.c
uint2rgb.m
Figure 2.5 Mosaicing code structure.
The interface between C and Matlab is organized by C files with special naming convention
(the name of the C file is the name of the function in Matlab). All files include mex.h header
file and contain the interface function mexFunction.
2.2 Calling conventions

General format for the C function definitions and calling convention as folows:
error_code = function_name([output][input&output][input]);
Almost all C functions provide as a return value the error code of execution. The error code
constants are listed below:
ERR_NONE
No errors.
10
ERR_WRONG_DIM
ERR_OUT_OF_MEM
ERR_SING_MTX
ERR_EMPTY_SET
ERR_NOT_FOUND
ERR_WRONG_COOR
-1
-2
-3
-4
-5
-6
The matrix is with wrong dimensions.

Out of memory error.
The matrix is singular.
Pixels set is empty.
Parameter is not found.
Image is with wrong coordinates.
The library consists of four common functions:

int optical_flow(double *u, double *v, int *fi2, int *errfi2, int *fo, int *fi, int width, int height,
double *gparam, int gparam_size);
int mutual_information(double *otfm, const int *fo, const int *fi, const int width,
const int height, const double *gparam, const int gparam_size,
const double *itfm, const int tfm_size);
int pixel_migration(double *otfm, const int *fo, const int *fi, const int width, const int height,
const double *gparam, const int gparam_size, const double *itfm,
const double *alpha, const int tfm_size);
int genmosaic3a(const char* cwd, const int histcor, const int Res, const int seq0,
const int band, const int coreg, const int brdsub, const int color,
const int poss);
Matlab calls the C code thought implementation of the mexFunction, with the following
calling convention:
void mexFunction(int nlhs, mxArray *plhs[], int nrhs, const mxArray *prhs[]);
The prototype of the function is declared in mex.h header file, where nlhs is the number of
expected outputs, plhs is array of pointers to output arguments, nrhs is the number of inputs,
prhs is array of pointers to input arguments.
3 input/output variable
Each common library functions has input/output variables as follows:
int optical_flow(double *u, double *v, int *fi2, int *errfi2, int *fo, int *fi, int width, int height,
double *gparam, int gparam_size);
Error code is returned as a numeric value.
The output is: u and v - transformation vectors, in the horizontal and vertical direction
respectively, fi2 - the transformed image, errfi2 - the error between the referenced image and
the transformed image.
The input is: fo - the reference image, fi - the observation image, width and height dimensions of the images, gparam - a vector with global parameters and gparam_size - the
dimension of the global parameters vector.
11
int mutual_information(double *otfm, const int *fo, const int *fi, const int width,
const int height, const double *gparam, const int gparam_size,
const double *itfm, const int tfm_size);
Theo output is: otfm - the image transformation parameters.
The input is: fo and fi - reference and observation image respectively, width and height dimensions of the images, gparam - a vector with global parameters, gparam_size - the
dimension of the global parameters vector, itfm - the initial image transformation parameter
and tfm_size - the size of the image transformation vector.
int pixel_migration(double *otfm, const int *fo, const int *fi, const int width, const int height,
const double *gparam, const int gparam_size, const double *itfm,
const double *alpha, const int tfm_size);
The output is: otfm - the image transformation parameter.
The input is: fo and fi - reference and observation image respectively, width and height dimensions of the images, gparam - a vector with global parameters, gparam_size - the
dimension of the global parameters vector, itfm - the initial image transformation parameter,
alpha - a balance vector in the Newtons iterative optimization and tfm_size - the size of the
image transformation vector.
Checking of the input variables form the Matlab interface is done by standard mex functions
mxGetN and mxGetN for dimensions and mxGetClassID for data type. Pointers to the input
variables are obtained by mxGetPr function, and proper type conversion is done where it is
necessary. The output variables are prepared by standard mex function
mxCreateDoubleMatrix.
3.1 Data types/structure

All images are presented in the library as a single dimension array of integers, int* in C, and
two additional variables for the width and height of the image. Working with single dimension
arrays makes the memory allocation and indexing operations faster.
All other parameters are presented as single dimension arrays of double numbers, double* in
C.
3.2 Functions list

int mxinv(double* mx, const int size);
Inverts matrix 'mx' with dimensions 'size' by 'size'.
12
double distance(const double *a, const double *b, const int size);
Calculate distance between two points in 2-D.
void print(const double *matrix, const int width, const int height);
Print matrix.
int mxaddlambda(double *matrix, const int size, const double lambda);
Add lambda to the main diagonal.
int mxnorm(double *matrix, const int size);
Normalize values of the matrix.
int refine(double *np, const double *alpha, const double *p, const double *GF, const double
*HF, const int size);
Refinement step in the Newton's Iterative Optimization.
int build_pyr(int **pyr, const int *img, int width, int height, int k);
Fill the pyramid with the image data.
int derivatives(double *dxI, double *dyI, double *dxF, double *dyF, double *F, double *dxxF,
double *dyyF, double *dxyF, double *dyxF, const int *img, const int width, const int height);
Calculate image intensity derivatives.
int high_gradient_set(int *set, const int *img, const int width, const int height, const double
percent, const int hist_size);
Estimate high gradient pixels set.
int S2xP(int *S1p, const int *S2, const int width, const int height, const int model, const
double *p, const int p_size);
Multiplies pixels set 'S2' by 'p'.
int form_GF(double *GF, const int model, const int *S1p, const double *dxF, const double
*dyF, const int width, const int height, const int GF_size);
Calculates gradient of the functional F(S1(p)).
int form_HF(double *HF, const int model, const int *S1p, const double *dxxF, const double
*dyyF, const double *dxyF, const double *dyxF, const int width, const int height, const int
HF_sqrt_size);
Calculates Hessian matrix of the functional F(S1(p)).
int newton(double *p, const double *alpha, const int p_size, const int *I1, const int *I2, const
int width, const int height, const int model, const double hgp_percent, const int hist_size,
const double epsilon, long iter);
Newton's Iterative Optimization over the functional F(S1(p)).
int gradient(double *grad, const int *img, const int width, const int height);
Calculate gradient of the image.
int compare(const int *Sa, const int *Sb, const int width, const int height);
Power of the set union between 'Sa' and 'Sb'.
13
double maxFSp(const int *S, const double *grad, const int width, const int height);
Evaluate the sum of the functional Sum( F(S(p)) ).
int randhrom(double *hrom, const int hsize, const double diff);
Initialize chromosome with random values in the range (-diif/2 , +diff/2).
int equals(const double *hroma, const double *hromb, const int hsize);
Compare two chromosomes for equivalence.
int sort(double **population, double *fitness, const int psize);
Sort all chromosomes in the population according their fitness.
int crossover(double **population, const int psize, const int hsize, const int copts);
Perform crossover in each chromosome in the population.
int mutation(double **population, const int psize, const int hsize, const int mpts, const double
mdiff);
Perform mutation in each chromosome in the population.
int slection(double **population, double *fitness, const int psize, const int hsize, const
double diff);
Select the chromosomes with the best fitness.
int ga(double *p, const int hsize, const int *I1, const int *I2, const int width, const int height,
const int model, const double hgp_percent, const int hist_size, const double epsilon, const int
iter, const int psize, const int copts, const int mpts, const double mdiff);
Applies Genetic Algorithm over the F(S1(p)) optimization problem.
int findparam(double *val, const char *str, const char *param);
Parse string to extract parameter value.
int fov(double *x, double *y, const double az, const double el, const double tilt, const double
h, const double azr, const double elr);
Transform field of view data into coordinates of four image corners.
int crossprod(double *c, const double *a, const double *b, const int size);
Cross product of vector 'a' by vector 'b'. The result is in vector 'c'. Working only with 3
elements vectors.
int fovpoint(double *p, const int p_size, const double az, const double el, const double tilt,
const double azr, const double elr);
Transform field of view data into unity vector.
int minamaxb(double *mina, double *maxb, const float *img2, const float *imgm, const
unsigned short *movl, const int size);
Find minimum and maximum intensity pixels in 'img2' or 'imgm' by using only 'movl'
overlapping area.
int formtbl(double *tbl, const int *H1, const int *H2, const int N);
14
Form bins matching table by using normalized accumulative histograms 'H1' and 'H2'.
int fitline(double p[2], const double *tbl, const int N);
Approximate the best fitting line according bins matching table.
int histnorm(int *H, const int N, const int norm);
Normalize histogram 'H'.
int cumhist(int *H, const float *img, const unsigned short *msk, const int size, const double a,
const double b, const int N);
Calculate accumulative histogram 'H'.
int histspec2(float *img2c, const float *img2, const float *imgm, const unsigned short *movl,
const int size, const int N);
Equalize histogram of image 'img2' according to histogram of image 'imgm' by using
overlapping area 'movl'.
int histspec2rgb(float *img2c, const float *img2, const float *imgm, const unsigned short
*movl, const int size, const int N);
Equalize histogram of image 'img2' according to histogram of image 'imgm' by using
overlapping area 'movl' in each color channel separetly.
int imtransform(unsigned int *img2, int xd[], int yd[], const double Tinv[9], const unsigned
int *img, const int width, const int height, const int xd2[2], const int yd2[2], const unsigned
int fill, const int sz2[2], const int color);
Transform image 'img' to 'img2' by using 'Tinv' transformation matrix and 'sz2' image
dimensions.
int loadsettings(const char* fname, double *val, const char *param);
Load setting from the settings file.
int mtrxAxB(double *c, const double *a, const double *b, const int aw, const int ah, const int
bw, const int bh);
Multiplication of matrix 'a' by matrix 'b'. The result is in matrix 'c'.
int UnitToQuadrilateral(double *A, const double *X);
Computes the 3-by-3 two-dimensional projective transformation matrix A that maps the unit
square ([0 0], [1 0], [1 1], [0 1]) to a quadrilateral corners (X(1,:), X(2,:), X(3,:), X(4,:)).
int construct_matrix(double *A, const double *U, const double *X);
Construct a 3-by-3 2-D transformation matrix A that maps the points in U to the points in X.
int projective(double *A, double *invA, const double *U, const double *X);
Calculate projective transformation matrix 'A' and inverse projective transformation matrix
'invA' by using four points coordinates 'U', which should match four points coordinates 'X'.
double rad(const double x);
Convert degrees in radians.
double vecnorm(const double *vector, const int size);
15
Calculate 'vector' norm. Size of the vector is 'size'.

int zerotomin(unsigned int *arr, const int size);
Replace zeros in the image with other value.
int shift4(unsigned int *trg, const int tw, const int th, const int d);
Cut the right edge of image 'trg'.
Cut the left edge of image 'trg'.
Cut the top edge of image 'trg'.
Cut the bottom edge of image 'trg'.
int placeIt4(unsigned int *trg, const int tw, const int th, const unsigned short *src, const int
sw, const int sh, const int xpos, const int ypos, const unsigned int mul);
Put on position ('xpos', 'yops') 'src' image into 'trg' image.
int placeIt3(unsigned int *trg, const int tw, const int th, const unsigned short *src, const int
sw, const int sh, const int xpos, const int ypos, const unsigned int mul);
int placeIt2(unsigned int *trg, const int tw, const int th, const unsigned short *src1, const
unsigned int *src2, const unsigned int *src3, const int sw, const int sh, const int xpos, const int
ypos);
Put on position ('xpos', 'yops') data from 'src1', 'src2' and 'src3' image into 'trg' image.
int placeIt1(unsigned int *trg, const int tw, const int th, const unsigned int *src, const int sw,
const int sh, const int xpos, const int ypos);
int mjmimm(unsigned short *mj, const int size, const unsigned short *mi, const unsigned
short *mm);
Combine masks 'mi' and 'mm' into mask 'mj'.
int extractIt(unsigned int *trg, const int tw, const int th, const unsigned int *src, const int sw,
const int sh, const int xpos, const int ypos);
Get from position ('xpos', 'yops') 'src' image of 'trg' image.
int maskImg(unsigned int *img, const unsigned short *msk, const int size);
Apply mask 'msk' on the image 'img'.
int formRmapRmapm(unsigned short *msk, const unsigned int *Rmap, const unsigned int
*Rmapm, const int size);
Update distance map according mosaic distance map.
16
int maskRmap(unsigned int *Rmap, const unsigned short *msk, const int size, const unsigned
int Rmax);
Update distance map according image mask and maximum distance in the distance map.
int mask(unsigned short *msk, const unsigned int *Rmap, const int size, const unsigned int
Rmax);
Form image mask according distance map and maximum value in the distance map.
int meshgrid(unsigned short *xx, unsigned short *yy, const int nc, const int nr);
Form x and y mesh grid.
int findRmax(unsigned int *Rmax, unsigned int *R, const int nc, const int nr);
Create distance map and finds the maximum value in it.
int update1(double Rmax, unsigned int *mmap, unsigned int *mos, unsigned int *mos2,
unsigned int *mid, unsigned int *Rmap, unsigned int *img2, unsigned short *msk, int width,
int height, int sz[2], int xd[2], int yd[2], double posid);
Initialize the mosaic and add the first image in it.
int update2(double Rmax, unsigned int *mmap, unsigned int *mos, unsigned int *mos2,
unsigned int *mid, unsigned int *Rmap, unsigned int *img2, unsigned int *img2c, unsigned
short *msk, int width, int height, int sz[2], int xd[2], int yd[2], double posid, int coreg, int
histcor, int color);
Add other images to the existing mosaic.
int limit(unsigned int *img2, unsigned int *Rmap, int sz[2], int xd[2], int yd[2], int *width, int
*height);
Limit image to fit in the mosaic dimensions.
int genmosaic3a(const char* cwd, const int histcor, const int Res, const int seq0, const int
band, const int coreg, const int brdsub, const int color, const int poss);
Main function for mosaic construction.
3.3 Memory allocations

The allocation and release of dynamic memory are done in the scope of the function where it
is needed. Allocation and release is done using standard C functions malloc, calloc, and free,
For the Matlab interface we used standard mex functions to allocate and release the dynamic
memory mxMalloc and mxFree. Matlab takes care of all memory allocated with mxMalloc,
but does not take care of the memory allocated with malloc and calloc, which memory should
be released by calling the function free.
17
4 Optical flow based registration

Optical flow is the displacement field related to each of the pixels, in an image sequence.
Such displacement field results from the apparent motion of the image brightness in time.
Estimating the optical flow is a fundamental problem in low level vision, and can undoubtedly
serve many applications in image sequence processing. There are many different methods to
estimate the optical flow. This report focuses on the method proposed by Pratikakis [36].
If we take a series of images in time, and there are moving objects in the scene, or perhaps
the camera itself is on some moving vehicle, useful information about what the image
contains can be obtained by analyzing and understanding the difference between images
caused by the motion. For example, given an image of a moving car, deciding which pixels in
the image represent motion can help to decide which pixels belong to the car, and which
pixels belong to the static background.
Studying the motion in detail, we can answer such questions as: How many moving objects
there are? Which directions they are moving in? Whether they are undergoing linear or
rotational motion? How fast they are moving?
From an image sequence, we calculate motion vector called the optical flow. For every pixel
a displacement vector V = (u, v) is found which says the speed and the direction of each pixel.
[30]
4.1 Theory
Co-registration is the term used for the registration of data from different sources in a unified
coordinate system. This section presents optic flow method for estimating motion in 2D gray
scale images.
The goal of this algorithm is to calculate the transformation between two images. For the
sake of clarity we define a reference image as fo and observation image as fi. The goal of the
co-registration algorithm is to find transformation parameters to transform the observation
image fi into the reference image fo in an optimal manner.
The general form for the transformation is in the form of a translation for each pixel in the
observation image fi:
u ( m, n) u m , n
(3.1)
v ( m, n ) v m , n
with (m, n) P and P is a set of all pixels in the image, u (m, n) is the translation in the
horizontal direction and v ( m, n) is the translation in the vertical direction for the pixel at
location. u m ,n and v m ,n are scalar values that need to be estimated. All possible motions are
included in this general form. Optical flow algorithm tries to estimate general image
transformation.
18
Other aspect of co-registration algorithm is the optimization criterion. Somehow there must
be a criterion to judge whether or not the estimated transformation is optimal. The most
common criterion is minimization of the sum square error Eq.(3.2), but it is also possible to
use correlation coefficient or the maximum a posterior probability.
SSE
(m, n) f i T ( m, n)
( m , n )P
(3.2)
where f i T (m, n) is the transformed observation image, using the transformation for each
pixel u (m, n) and v ( m, n) .
Image interpolation is necessary for smooth transformation of the observation image fi so
that it can be matched with the reference image fo, after estimation of the pixel transformation
function.
5 Dense optic flow

A 6-parameter affine motion estimator in the frequency domain is described by Kruger [26].
The advantage is that it solves the translation and the linear component separately. However,
it can only be applied to local regions. Therefore, it is necessary to divide the two images into
blocks and apply the algorithm on each block, followed by the accumulation of the results for
the whole image.[45] A multi-scale and multi-resolution motion estimator has been developed
at INRIA [32][36].
The formulation of the co-registration problem as optical flow hypothesis is introduced by
Horn and Schunck[16] and leads to the minimization of the cost function:
2
U ( w, f ) f ( s, t ) ws f t ' ( s, t )
sS
ws wr
(3.3)
s , r C
where s is a pixel from the set S (pixel lattice), t is the temporal index of the images, f is the
luminance function, f is the gradient of the luminance function, f t ' is time derivative, w
is the expected displacement field, C is the set of neighboring pairs and controls the balance
between the two energy terms.
Because the cost function does not make any difference between relevant data and
inconsistent data, and it is sensitive to noise, therefore, robust M-estimators have been
introduced in [1]. The main benefit of robust M-estimator is the semi-quadratic formulation.
*
U ( w, s , sr , f ) s f ( s, t ) ws f t ' ( s, t ) 1 s
sS
w
sr
s , r C
wr
2 sr
(3.4)
where s and sr are auxiliary variables acting as weights.

For large displacement multi-resolution and multi-grid are developed. A pyramid of volumes
{fk} is constructed by successive Gaussian smoothing and subsampling. For the next
19
resolution levels, only an increment is estimated to refine estimate obtained from the previous
level Eq.(3.5).
2
*
k
k
k

U k (dw k , k , k , f k , w) sk f k ( s w, t 2 ) dwsk f k ( s w, t 2 ) f k ( s, t1 ) 1 sk
sS

w
k
sr
s , r C
^k
s
dwsk wr^ k dwrk
2 srk
(3.5)
At each level of resolution, a multigrid minimization based on successive partitions of the

initial image is achieved. For each rectangle of given level, a 6-parametric increment field is
estimated. The result over the grid level is a rough estimate of the desired solution, and it is
used to initialize the next grid level. This hierarchical minimization strategy improves the
quality and the convergence rate.
5.1 Implementation
Implementation of the Optic Flow algorithm is provided by Pratikakis [36] and it is
organized in C code files and header files. We developed warping function to provide
interface between C and Matlab. We used Matlab for testing and analyzing the experimental
results.
5.2 Experiments and discussion

In this work we took Yosemite sequence, representing well known motion and taxi sequence.
The generated terrain images 256x256 with 256 levels of intensity are used in subsequent
estimation of the motion parameters. Because the parameters of the motion are known in
advance it is possible to compare the estimated values with the correct ones. The comparison
is done between frames with number 00 and 01, which can be seen on the Fig.(3.1) and Fig.
(3.2).
We can notice easily that the estimated motion is stable and similar to the expected one.
Clouds, at the top side of the image, are moving from left to right. The hill is getting near
from the center to the right-bottom corner.
With the taxi sequence pictures Fig. (3.3) we do not know the motion in advance, but we
have general idea about it. There is a taxi taking right turn, a car, on the left side, moving from
left to right, a bus, on the right side, moving from right to left and a human walking on the
pavement.
The needle map and the resampled image in Fig.(3.5) shows us the moving objects in the
center, on the left and the right side of the picture.
The Optic Flow Method is proper to estimate motion when the changes in the picture are not
so big and the intensity of the pixels remains the same. In more complex transformation,
20
combining translation and light intensity changes, the optical flow dose give giving
satisfactory results.
Figure 3.6 Computer generated terrain images. a)Fist frame. b)Second frame.
Figure 3.7 Estimated motions for Yosemite sequence. a)Estimated motion between first and second frame.
b)Expected motion that we know a prior.
Figure 3.8 Taxi sequence images. a)Fist frame. b)Second frame.
21
Figure 3.9 A)The transformed image, the first fame to the second frame in the taxi sequence. B) Needle
map of the estimated motions for the taxi sequence
During the experiments, we found the lateral effect of the image blocking, which can be
interesting as a future work and research.
22
6 Pixel migration based registration

We offer an implementation of the pixel migration method, introduced by J. Keller [24]. The
registration is achieved without applying a similarity measure to the input images. Intensity
invariant geometrical alignment is achieved while using robust global optimization. The
algorithm aligns the set of pixels having large gradient magnitudes in both images. This set is
invariant under most intensity changes and geometrical transformations. The alignment is
conducted using iterative optimization; thus the high-gradient pixels of one image migrate to
their corresponding locations in the second image. The alignment is carried out using a single
image (the second images pixel set is used solely for initial input). Thus, the convergence
properties of the proposed algorithm depend entirely on a single image which can be chosen
to be the less noisy input image. This property makes the proposed algorithm especially
suitable for multi-sensor and multi-modality registration where conventional image
registration techniques may fail [17].
6.1 Theory
We denote I1(x,y) and I2(x,y) as images which have some common overlap, the idea of pixel
migration is to register two images by using a set of high gradient pixels and maximization of
functional F(S1(p)):
F S1 p
I 1 ( x i , y i )
(4.6)
( xi , y i )S1 p
Where S1 is a set of pixels in I1 and || . || is a Euclidean norm.

Usually pixels with high gradient magnitudes correspond to primary structures within the
image and we can use it to formulate proper optimization function. We denote S2 as a set of
coordinates of high gradient edge pixels detected in I2. We use affine and projective
transformation models to formulate motion between the two images. Both models are
presented with global parameter p, which is a vector. Usually as a parameter we choose the
size of S2 around 20% of the total pixels in I2.
Global alignment is achieved by maximizing the functional F(S1(p)), defined as:
p* arg max F S1 p
(4.7)
while the set S1(p) is given by:

S1 ( p ) T ( S 2 , p )
(4.8)
where S1(p) is the set of S2 coordinates projected according to the transformation function T(.)
Eq.(4.5)-(4.7) and transformation parameter vector p. The algorithm starts with p0 which is an
initial estimate of motion parameters.
23
Hence, p*, the solution to Eq. (4.2), maps the set S1(p*) to the set S2 and since the motion is
assumed to be global, p* is an implicit solution to the image registration problem when no
explicit similarity measure is used.
7 Iterative optimization
Solution to Eq. (4.2) will be obtained as an iterative optimization. The optimization is
conducted using I1 solely.
F(S1(p)) is non-linear with respect to p. Hence, the transformation parameters are computed
using Newtons iterative optimization method [9] where the iterative refinement equation is
given by:
p n 1 p n ( H pn ) 1 p Fn
(4.9)
where p Fn is the gradient vector of F with respect of p at iteration n, H pn is the Hessian

matrix of F with respect to p at iteration n and pn is the parameter vector at iteration n.
Let p Fn xi , y i and H pn xi , y i be the gradient and Hessian of F(S1(p)), respectively,
evaluated at a pixel xi , y i . Then p Fn xi , y i and H pn xi , y i are accumulated over the
complete pixel set S1(p).
To obtain S1(p) on each step we need to transform the coordinates of the pixels in S2 using
transformation function T(.) Eq.(4.3) defined as follows Eq.(4.5)-(4.7):
x
X Affine
0
y
0
1
0
0
x
0
y
0
1
(4.10)
or
x
X Pr ojective
0
y
0
1
0
0
x
0
y
0
1
x2
xy
xy
y2
(4.11)
where shortly we denote XAffine or XProjective as X:

x
y Xp

(4.12)
Because the affine and projective models are linear with respect to p for these models, the
gradient vector and Hessian matrix can be expressed as:
p Fn
T
i
( xi , yi )S1 ( p )
H pn
T
i
( xi , y i )S1 ( p )
( x , y ) Fn ( xi , y i )
H (nx , y ) ( xi , y i ) X i
(4.13)
(4.14)
24
where
F ( xi , y i )
( x , y ) Fn ( xi , y i )
x
F ( xi , y i )
(4.15)
and
2 F ( xi , y i )
x 2
H (nx , y ) ( xi , y i ) 2
F ( xi , y i )
yx
2 F ( xi , y i )
xy
2 F ( xi , y i )
y 2
(4.16)
are the gradient and Hessian, respectively, evaluated at xi , y i at iteration n with respect to
the Cartesian coordinate system.
We iterate Eq. (4.4) until either of the following convergence criteria is met:
1. A maximal (predefined) number of iterations is reached.
2. The update of the parameters in Eq. (4.4) becomes smaller than a predefined size
:
p n 1 p n
3. The value of
(4.17)
p n 1 p n
gets bigger than in the previous iteration.
8 Genetic Algorithms
Another approach to provide solution for Eq.(4.2) is to find the maximum using genetic
algorithms.
25
Initialization
p0
Optimization
loop
Crossover
Mutation
Selection
p*
Figure 4.10 Genetic algorithm, basic diagram.
Genetic algorithm Fig. (4.1) begins with initialization. After that in a loop we repeat many
times crossover, mutation and selection. End of the optimization cycle is after defined number
of epochs or satisfying convergence.
If we solve a problem, we usually look for some solution which is the best among others.
The space of all feasible solutions is called search space, which in our case is presented of all
possible values for the global transformation parameter p. Each point in the search space
represents one possible solution. Each possible solution can be "marked" by its value for the
function F(S1(p)).
Algorithm begins with a set of solutions, different p values as chromosomes encoded as real
numbers, called population. Solutions from one population are taken and used to form a new
population. This is motivated by the hope, that the new population will be better than the old
one. Solutions are selected to form the new population according to their fitness, measured as
value of F(S1(p)) - the more suitable they are the more chances they have to reproduce.
Crossover operates on selected genes from parent chromosomes and creates new offspring.
The simplest way to do that is to choose randomly some crossover point and copy everything
before this point from the first parent and then copy everything after the crossover point from
the other parent.
After a crossover is performed, mutation takes place. Mutation is intended to prevent falling
of all solutions in the population into a local optimum of the solved problem. Mutation
operation randomly changes the offspring resulted from crossover. In our case we change a
few randomly chosen values.
26
There are many methods in selecting the best chromosomes, but in our case we just keep the
best fitted solutions. [34]
9 Multiscale extension
To improve the convergence properties of the iterative algorithm a multi-resolution pyramid
is constructed. The alignment algorithm starts at the coarsest resolution scale of the pyramid,
in a coarse-to-fine approach. At each resolution scale, the precisely described calculation is
conducted where the results of the calculation at each scale serve as an initial guess for the
finer scale. Finally, when the procedure stops at the finest scale, the final transformation
parameter is obtained.
p{0}
p{1}
p{2}
Figure 4.11 Multiresolution pyramid is used to estimate initial values for p.
We should be careful in using coarse-to-fine refinement process, because on the coarse level
it is possible to have to less information, which can be critical for the pixel migration. Usually
we use up to three levels for the multiresolution extension.
9.1 Implementation
Algorithm presented on the Fig.(4.4) begins with estimation of the high gradient pixels in
image I2. By using histogram based approach of image gradients we estimate the proper high
gradient pixels percent Fig.(4.3). High gradient pixels percent serves as a parameter to the
implemented algorithm.
27
Figure 4.12 High gradient pixels set - a) the original image, b) 5% as high gradient pixels, c) 10% as high
gradient pixels, d) 15% as high gradient pixels.
The next step is to calculate the derivatives of the intensity function of image I1. Second and
third order derivatives are used to calculate the gradient and Hessian of F(S1(p)) Eq.(4.12),
Eq(4.11). The set of coordinates of high gradient pixels in I2 is used to calculate the set of
coordinates of high gradient pixels S1(p), using Eq.(4.5) or Eq.(4.6) and Eq.(4.7).
pn 1 pn ( H pn ) 1 p Fn
(4.18)
We calculate the gradient vector and Hessian matrix Eq.(4.8) and Eq.(4.9) on each step of
the Newtons iterative optimization. The Hessian matrix is inverted and we apply refinement
Eq.(4.13). We modified slightly Eq.(4.6), by introducing additional parameter vector for
better control over the iterative refinement.
28
I2
I1
p0
For
each
pixel
Estimate
Calculate
S2
I1x, I1y,
I1xx, I1yy, I1xy,
I1xxx, I1yyy, I1xxy,
I1yyx
XAffine
XProjective
or
Multiply
S1(p0)
pFn
Inv(Hpn)
Hpn
pn+1 = pn + Inv(Hpn)* pFn
S1(pn) =
T(S2,pn)
For n=0
Take p0
||pn+1 - pn|| <
n=n+1
Yes
END
Figure 4.13 Diagram for Newtons iterative optimization.
To compare the efficiency of pixel migration method with Newtons method we developed
the same pixel migration concept with genetic algorithms as an optimization approach.
Because of the probabilistic nature of the genetic algorithms they should be used with some
idea for the probability distribution of the problem. This is necessary for the choice of the
29
genetic algorithms parameters. The convergence can be controlled by parameters as follows:

population size, crossover points, mutation points, mutation tolerance, and number of epochs
(generations). As initial population we chose p0 (initial global transformation parameter
vector) and values near to it.
I2
I1
Calculate
I1x, I1y
p0
For
each
pixel
Estimate
S2
XAffine
XProjective
or
Multiply
S1(p0)
Population
Crossover
n=n+1
Mutation
S1(pn) = T(S2,pn)
Selection
F(S1(pn))
||pn+1 - pn|| <
or n == max n
Yes
END
Figure 4.14 Diagram for genetic algorithm optimization.
30
9.2 Experiments and discussion

Case 1: For the first experiment we took the image of Lena, 480x480, with 256 levels of
intensity, as a reference image. The observation image was obtained applying affine
transformation over the same image.
Newtons iterative optimization converges, but we should choose carefully values for the
parameter . Newtons iterative optimization is proper only for local optimization and because
our algorithm is very sensitive of the initial values.
Figure 4.15 Lenas picture. Reference image is on the left side. Observation image is on the right side.
For Lenas images first we applied the pixel migration method with Newtons iterative
optimization, as optimization approach, and three levels for the multiresolution pyramid Fig.
(4.7) and Fig.(4.8).
In the experiment we used affine transformation model and initial value p0 = [1 0 0 0 1 0]T.
Motion estimation is pretty good, which can be seen on the Lenas shoulder, the towel in the
right-bottom corner and the mirror. Estimated value is p = [0.99745 0.0064354 -0.069282
-0.0098789 0.97872 0.62621]T
31
Figure 4.16 Lenas transformed image, with Newtons iterative optimization.
Figure 4.17 Iterative optimization co-registration error of Lenas images. Left side - the reference image in
the red channel and the observation image in the green channel. Right side - the reference image in the red
channel and the transformed image in the green channel.
After that we applied the pixel migration method with genetic algorithm, as optimization
approach, and three levels for the multiresolution pyramid. On Fig.(4.9) is the transformed
image, on Fig.(4.10) is the comparison between the reference image and the transformed imag
and on Fig.(4.11) is the performance graphic.
In the experiment we used 350 epochs, affine transformation model and initial value p0 = [1
0 0 0 1 0]T. Motion estimation is pretty good, which can be seen on the Lenas shoulder, the
towel in the right-bottom corner, the hat and the hair. Estimated value is p = [1.0274 0.0
5.8269 -0.055527 1.0008 6.4521]T
32
Figure 4.18 Lenas transformed image, with genetic algorithm.
Figure 4.19 Genetic algorithm co-registration error of Lenas images. Left side - the reference image in the
red channel and the observation image in the green channel. Right side - the reference image in the red
channel and the transformed image in the green channel.
33
Figure 4.20 Performance of the genetic algorithm optimization with Lenas images F(S1(p)).
Case 2: In the second experiment we used image of the truck in the desert, taken by plane,
460x460, with 256 levels of intensity Fig.(4.12).
We used affine transformation model, three levels of multiresolution pyramid and genetic
algorithm. We have a good estimation of the motion, which is visible on the performance
evaluation image Fig.(4.14). Dominant motion is connected with positions of the road and the
truck on it.
Figure 4.21 Picture of the truck in the desert. Reference image is on the left side. Observation image is on
the right side.
34
Figure 4.22 Transformed truck image, by transformation parameter estimated with Genetic algorithm.
Figure 4.23 Co-registration error of the truck images. Left side - the reference image in the green channel
and the observation image in the red channel. Right side - the reference image in the red channel and the
transformed image in the green channel.
35
Figure 4.24 Performance of genetic algorithm optimization over the truck images. Top - F(S1(p)). Bottom difference ||pn+1 - pn||.
Case 3: In the third experiment we used real area images, taken by moving plane from
average flying height, 256x256 with 256 levels of intensity Fig.(4.16).
We used affine transformation model, three levels of multiresolution pyramid and Newtons
iterative optimization. Convergence of the algorithm can be seen on Fig.(4.19).
Figure 4.25 Real area images. Reference image is on the left side. Observation image is on the right side.
36
Figure 4.26 Transformed area image, by transformation parameter estimated with iterative optimization.
x 10
F( S1(p) )
F( S1(pn) )
6
4
2
0
1.5
2.5
1.5
2.5
3
3.5
Iterations
Convergence
4.5
4.5
||Pn+1 - Pn||
15
10
3
Iterations
3.5
Figure 4.27 Performance of the Newtons iterative optimization, applied on the real area images. Top F(S1(p)). Bottom - difference ||pn+1 - pn||.
Optimizing Eq.(4.1) leads to a solution, but calculating the objective function Eq.(4.1) is too
much time consuming, which makes genetic algorithms to converge in not reasonable amount
of time.
37
10 Mutual information based registration

The use of mutual information (MI) as a similarity measure for image registration problems
was first proposed by two groups of researchers in the Catholic University of Leuven [50],
Belgium and MIT in 1995. It has made vital contributions to many different disciplines like
physics, mathematics, economics, and computer science. Maximization of mutual information
has been applied successfully for many remote sensing applications.
Mutual information has its roots in information theory, it is a measure of statistical
dependency between two data sets. It does not depend on any assumption on the data, does
not assume specific relations between intensities in different modalities and it is particularly
suitable for registration of images from different modalities.
The MI registration criterion states that the MI of the image intensity values of
corresponding pixel pairs is maximal if the images are geometrically aligned. Because no
assumptions are made regarding the nature of the relation between the image intensities in
modalities, this criterion is very general and powerful and can be applied automatically
without prior segmentation on a large variety of applications.
10.1
Theory
The MI of two random variables A and B is defined by:

I ( A, B ) PA, B (a, b) log
a ,b
PA, B (a, b)
PA (a ) PB (b)
(5.19)
where PA a and PB b are the marginal probability mass function, and PAB a, b is the
joint probability mass function. MI measures the degree of dependence of A and B by
measuring the distance between the joint probability PAB a, b and probability associated
with the case of complete independence .
MI is related to entropies by the relationship as:
I ( A, B) H ( A) H ( B) H ( A, B)
(5.20)
where H(A) and H(B) are the entropy of A and B and H(A, B) is their joint entropy.
Considering A and B as floating image (F) and reference image (R) respectively, the MI
based image registration criterion postulates that the image is registered when I(F, R) is
maximum.
The entropy and joint entropy can be computed from,
H ( F ) PF ( f ) log PF ( f )
f
38
H F PR r log PR r
(5.21)
H ( F , R) PF , R ( f , r ) log PF , R ( f , r )
f ,r
To employ mutual information as a similarity measure, we need to utilize the concept of the
two dimensional histogram of an image pair, the joint histogram. The joint histogram h is the
joint histogram of the image pair. It is a 2D matrix given by:
h 0,0
h1,0
h
...
h M 1,0
h 0,1
h1,1
...
h M 1,1
(5.22)
...
...
h 0, N 1
h1, N 1
...
...
... h M 1, N 1
The value of h(f, r) ( f [0, M 1] , r [0, N 1] ) is the number of pixel pairs having
intensity value f in the first image and intensity value r in the second image. M and N are the
number of gray levels used in the images F and R respectively. Typically, M=N=256 for 8 bit
images.
The joint probability mass function used in the calculation of MI of an image pair is then
obtained by normalizing the joint histogram of the image pair as:
PF , R f , r
h f , r
h f , r
(5.23)
f ,r
From the joint probability mass function, we obtain the two marginal probability mass
functions directly as:
PF f PF , R f , r
r
(5.24)
PR ( f ) PF , R ( f , r )
f
From the above equations, we see that the joint histogram of an image pair is the only
quantity required to calculate for the mutual information. As a result, the quality of the
estimated jointed histogram solely determines the accuracy of this method.
Generally, the joint histogram is estimated by a two-step procedure. Denote the two images
that need to be registered as F, the floating image on which a geometric transformation is
applied, and R, the reference image that is interpolated.
1. The first step is to estimate the intensity values in R at every transformed grid point
of F by employing an intensity interpolation algorithm.
2. The second step is then to determine the joint histogram from the overlap of the
floating image and the interpolated reference image.
39
There are two methods popularly adopted to estimate the required joint histogram: linear
interpolation and partial volume interpolation (PVI). However, it is found that the
performance of the existing joint histogram estimation algorithms is limited due to a
phenomenon called interpolation-induced artifacts that introduce spurious local maxima.
To overcome this problem, estimation named generalized partial volume estimation (GPVE)
is used. This method generalizes the PVI algorithm to overcome the artifact problem. In
general, a kernel function is employed for the estimation of the joint histogram of two images.
It can be shown that the PVI algorithm is a special case of the generalization. It corresponds
to the use of the first order B-spline as the kernel function. By choosing the 2nd order or the
3rd order B-spline as the kernel function, the artifacts can be removed satisfactorily.
10.2
Nonlinear conjugate gradient method
Consider the unconstrained optimization problem
minn f ( x)
(5.25)
xR
where f is a smooth function and its gradient is available. Conjugate gradient methods are a
class of important methods for solving (5.7), especially for large scale problems, which have
the following form:
x k 1 x k k d k
(5.26)
g k
d
gk k d k1
(5.27)
where g k f ( x k ) , k is a stepsize obtained by a one-dimensional line search and k is

a scalar. The strong Wolfe conditions, namely,
f ( x k k d k ) f k k g kT d k
T
(5.28)
T
k
g ( x k k d k ) d k g d k
(5.29)
where 0 1 , are often imposed on the line search (in this case, we call the line search
the strong Wolfe line search). The scalar k is chosen so that the method (5.8)-(5.9) reduces
to the linear conjugate gradient method in the case when f is convex quadratic and exact line
search g ( x k k d k ) T d k 0 is used.
For general functions, however, different formulae for scalar k result in distinct nonlinear
conjugate gradient methods. Several famous formulae for k are the Fletcher-Reeves (FR),
Polak-Ribiiere-Polyak (PRP), and Hestenes-Stiefel (HS) formulae, which are given by
40
FR
k
PRP
k
g k 1
kHS
gk
(5.30)
g kT y k 1
g k 1
(5.31)
g kT y k 1
d kT1 y k 1
(5.32)
respectively, where means the Euclidean norm and y k 1 g k g k 1 . If the condition

imposed on in (5.11) is only that 1 , neither of the above three famous nonlinear
conjugate gradient methods with the strong Wolfe line search can ensure a descent search
direction even if f is quadratic.
The conjugate descent (CD) method of Fletcher, where
CD
k
gk
(5.33)
d kT1 y k 1
ensures a descent direction for general functions if the line search satisfies the strong Wolfe
conditions (5.10)-(5.11) with 1 . But the global convergence of the CD method is proved
only for the case when the line search satisfies (5.10) and
g kT d k ( g ( x k k d k ) T d k ) 0
(5.34)
For any positive constant , the conjugate descent method with k satisfying (5.10) and
g kT d k ( g ( x k k d k ) T d k ) g kT d k
(5.35)
need not converge.
10.3
Implementation
After introducing the main theories we used in our project, the flow chart of the MI based
registration system is drawn in Fig. 5.1.
41
Figure 5.28 Flow chart of the MI based registration system.
There are two input images: reference image R and floating image F. With the initial set of
transformation parameters, the joint histogram of the two input images is estimated and then
the mutual information is calculated. There is an optimization method to estimate whether the
mutual information is at a maximum. If not, update the transformation parameters set and go
to the joint histogram estimation step; if the maximum of the mutual information is achieved,
it means the reference image and the floating image are registered, the calculation stops.
42
10.4
Experiments and discussion
For the experiment we used agricultural area pictures from different time periods Fig. (5.2).
We did the experiment using rotation and translation model.
Figure 5.29 Agricultural area pictures. Left 09/07/94. Right 21/09/92.
The nonlinear conjugate gradient method performs slowly, because we calculate many times
partial derivatives of our similarity function.
Figure 5.30 Agricultural area transformed image.
In Fig. (5.3) is the transformed image, but to achieve convergence of the co-registration
process we need to try many different values of the parameters and k .
43
11 User interface
The purpose of the Graphic User Interface for the Image Co-registration Library is to
provide proper facilities for testing of the implemented modules and the developed executable
code. Current GUI is a good point for initial testing and some useful results in the testing of
the whole library.
11.1
Overview
The GUI runs on Matlab environment under Windows or Linux operating systems. Because
of the machine dependences of the C code, the C code should be precompiled under operating
system before being used with the GUI.
The GUI should be started in the proper working directory from the Matlab command line. It
is necessary because the GUI expects to find proper executable code (DLL files) in the
specific subdirectories. Because of the path dependences the file structure of the library
should remain strictly unchanged.
11.2
Tutorial
Here is a sequence of steps that takes you through the basics of the GUI. When you are told
to click on a button, this will mean click once with the left mouse button unless specifically
stated otherwise.
To start change the Matlab working directory to the imreg_GUI directory and enter the
command imreg_UI. A window should appear on your screen that looks like in Fig.(6.1).
44
Figure 6.31 User interface main working screen.
The layout consists of seven panels in the main window, which is not resizeable:
* Working directories/files
* Optic flow method
* Mutual information method
* Pixel migration method
* Visualization area for the first image
* Visualization area for the second image
* Status line
To close the GUI you can use the standard window X button or from the menu by choosing
File->Exit, the quick combination Alt+F4 is also working. Closing of the GUI can be problem
if the working directory of Matlab is different then imreg_UI. If this is the case you should
change the working directory to imreg_UI.
12
Input of images
In the working directories/files panel you should browse both images (referenced and
observation) which should appear in the visualization areas.
45
Click on the Browse button to open standard Open File Dialog. Change to the proper
directory with the images, select one that you need.
13
Selection of registration methods and parameters
After the images were chosen you may want to review and change the parameters for each
method. You can do that just by click on the Options button, for the different methods.
In the options dialogs you can choose different values and different models for the coregistration methods. The options dialogs can be closed by clicking on the OK or on the
Cancel button. If you close the dialog with OK all options will be saved in pure text options
file. When you open the options dialog next time the parameters will be loaded in the form
automatically. If you close the dialog with Cancel or standard window X button changes will
not be saved.
Figure 6.32 Optic Flow Options Dialog.
46
Figure 6.33 Mutual Information Options Dialog.
47
Figure 6.34 Pixel migration options dialog.
To choose and start the calculation press Start button in the panel of the desired coregistration method. There is not visual indication that the calculation is in progress, except
the Busy message in the main Matlab window. The progress visual indication is not possible
because of the C function calling (DLL executable code), which occupies all the system
resources during its execution. During the calculation GUI is also blocked because Matlab
dose not offer multithreading in the graphic user interface update.
14
Visualization of results
For each co-registration method visualization routines are different and the results are
available immediately after the calculation in standard Matlab figures. You can use all the
functionality provided by the standard Matlabs figures, including Resize, Save As, Print, etc.
In the status line you can observe for the error messages. Also the main Matlabs window
will display all critical and non-critical errors. The non-critical errors are displayed in the
Matlabs console with blue color and the calculation is not stopped. The critical errors are
displayed with red color, describing the error. After the appearance of the critical error the
calculation is stopped and the current global working variables are still accessible in the
Matlabs console with commands who and whos. Matlab takes care to free all the memory
reserved with the function mxMalloc but dose not take care for the memory allocated with the
C functions malloc, calloc and realloc. Because of the specific memory handling with the
48
dynamic memory in C special efforts were taken for strict error handling control (out of
memory, illegal arithmetical operation and etc.). The proper error code is used for the return
value in almost all C function.
15
Limitations of user interface
* The GUI is blocked because of the way in which Matlab invokes C functions
implemented in a DLL.
* Non-critical runtime errors - usually appear during the visualization of the figures in
Matlab.
* Critical runtime errors - usually out of memory or arithmetic errors during the C
code execution.
* Error message are sent as return values in the C functions.
* Equal dimensions between the observation and the referenced image.
49
16 Mosaicing
Our work of building mosaics is based on the previous project in ETRO-VUB named
ClearFast. The goal of our work is to provide C implementation of the existing Matlab
code.
ClearFast is an abbreviation of Concept for Low-risk Efficient Area Reduction Based on
the Fusion of Advanced Sensor Technologies. The basic idea in the project is to detect
landmines by taking infrared pictures during 24 hours of the landmine field.
The mosaicing procedure is primarily used to provide an overview of the surveyed area and
to locate the detected suspected objects. For a scanning mission, the mosaicing procedure
combines all the acquired images into a large mosaic image. The image mosaic can be seen as
an aerial photograph taken from very large height.
16.1
Background
Through your eyes, you see the world in wide-angle; your view extends more then 180
degrees from the front to the periphery of your vision. Through a camera, however, you see
the world through the small box of the viewfinder, limited to this partial view and missing the
larger picture. Mosaic images, constructed from a series of overlapping camera images, which
tile over a scene, permit you to see the whole picture again. Mosaics break the constraint of a
single cameras view to allow you to visualize an entire scene with a much larger field of
view.
Creating mosaic images require first registering a set of images to compute orientation
between adjacent frames, and then merging them together into a seamless composite. This can
be fairly straightforward for stitching a few frames together. Or it may require stitching a
wide-angle horizontal strip through 360 degrees and then matching the two ends into a
continuous circle, or fitting together a 2D collection of images, or even creating a 2D fullcircle spherical view of an environment.
The quality of the registration is dependent on the complexity of the imaging process, and
can require a range of models to accurately map from one image to the next, from simple
translation and rotation to affine to projective. The images may also require various forms of
pre-processing to remove lens artifacts (i.e., barrel distortion), and to account for lens
characteristics (i.e., focal length). The result of this pre-processing and analysis, then, is a set
of alignment mappings between adjacent images, computed at sub-pixel resolution.
The compositing process then creates a final mosaic image that is stitched together from
pieces of the set of images. Compositing requires intelligent selection of portions of the
original image, and careful blending around each piece to avoid visible seams, either from too
sharp a transition or excess blurring. Other related processing can be performed, including
balancing the exposure changes between frames or enhancing the original resolution by using
the additional data in the overlapping regions. [52]
50
17
Why we need mosaic construction in ClearFast project?
Both cameras (IR and RGB) used in ClearFast could provide us pictures with wide field of
view but without high resolution in the same time. The high resolution is very important for
us because of the relatively small objects that we want to detect. In order to achieve that the
experiment was done by taking many small pictures and after that combining them in a big
mosaic. Pictures were taken in 20-30 minutes cycles which involve light intensity changes
and camera position changes. When we construct the mosaic we need to take care of these
changes.
17.1
Theory and formulations
The usual approach of mosaic building is by constructing a strip, when the motion is in one
direction. If there are motions in two directions, the mosaic can be build up by first generating
the strips and then mosaicing the strips. Common disadvantage of this method is the
accumulative error and not proper alignment of the strips.
In our project the mosaics are constructed from model with two directions motions, as it is
shown on the Fig.(7.1).
tilt
azimuuth
elevation
h
center
azimuuth
Figure 7.35 Camera model for the ClearFast project.
The mosaic is constructed by changes in the azimuth and the elevation, together. We have
also an undesired tilt angle (fixed), which is an effect of the equipment, but we need consider
it, where h is the distance of the camera from the ground.
All images are taken with idea of the parameters azimuth, elevation, height and tilt, which
means that we have general idea about the positions of each image in the final mosaic.
The mosaic construction procedure is summarized as follows:
51
1. Image back-projection onto ground surface;

2. Determine overlap with the existing mosaic;
3. Shift correction (not implemented);
4. Histogram equalization of the new added image;
5. Update mosaic.
17.1.1
Ground projection
Because we can estimate the coordinates of the four image corners, using the azimuth and
the elevation we can easily calculate the projective transformation matrix and to project the
image on the surface.
z
p
h
R
zg
xg
x
yg
y
Figure 7.36 Vector p is the unity vector pointing from the focal point onto the ground.
For each of the four corners we obtain the coordinates of the vector p:
p
p1t p 2t tan(elr ) p3t tan(azr )
(7.36)
p1t p 2t tan(elr ) p3t tan(azr )
Where:
p1t M p1
p 2t M p 2
t
3
t
1
p p p
(7.37)
t
2
And t is a tilt angle, M is transformation matrix correcting the tilt.

cos(t )
M 0
sin(t )
0
1
0
sin(t )
0
cos(t )
(7.38)
52
Also
cos(el ) cos( az )
p1 cos(el ) sin( az )
sin(el )
(7.39)
sin(el ) cos( az )
p 2 sin(el ) sin( az )
cos(el )
(7.40)
Where el is elevation angle and az is azimuth angle.

We normalize p coordinates in order to
p 1:
xp

p yp
zp

(7.41)
h
zp
(7.42)
Where R is the distance of the camera to the point on the ground.

x p 0

y p 0 R p
z p h

And 0
(7.43)
is the camera position.
With the coordinate of the image corners we calculate the transformation matrix T.
T Tu1Tx
(7.44)
Where Tu and Tx are the transformation matrices in the domain: U - the domain of the
original image and X is the domain of the image back projected onto the ground.
x p 2 x p1 t13u x p 2
Tu x p 4 x p1 t 23u x p 4
x p1
x
13
x
23
x g 2 x g1 t x g 2
Tx x g 4 x g 1 t x g 4
x g1
y p 2 y p1 t13 y p 2 t13u
u
y p 4 y p1 t 23 y p 4 t 23
y p1
1
x
13
x
23
y g 2 y g1 t13 y g 2 t
y g 4 y g1 t 23 y g 4 t
y g1
1
53
(7.45)
With ( x pn , y pn ) and ( x gn , y gn ) as x and y coordinates of point n of the corners:
t13u
( x p 1 x p 2 x p 3 x p 4 )( y p 4 y p 3 ) ( x p 4 x p 3 )( y p 1 y p 2 y p 3 y p 4 )
( x p 2 x p 3 )( y p 4 y p 3 ) ( x p 4 x p 3 )( y p 2 y p 3 )
u
t 23
( x p 2 x p 3 )( y p 1 y p 2 y p 3 y p 4 ) ( x p1 x p 2 x p 3 x p 4 )( y p 2 y p 3 )
( x p 2 x p 3 )( y p 4 y p 3 ) ( x p 4 x p 3 )( y p 2 y p 3 )
(7.46)
t13x
( x g 1 x g 2 x g 3 x g 4 )( y g 4 y g 3 ) ( x g 4 x g 3 ) ( y g 1 y g 2 y g 3 y g 4 )
t23x
( x g 2 x g 3 )( y g 1 y g 2 y g 3 y g 4 ) ( x g 1 x g 2 x g 3 x g 4 )( y g 2 y g 3 )
( x g 2 x g 3 )( y g 4 y g 3 ) ( x g 4 x g 3 )( y g 2 y g 3 )
( x g 2 x g 3 ) ( y g 4 y g 3 ) ( x g 4 x g 3 ) ( y g 2 y g 3 )
For each pixel x in the image I we resized the coordinates according the desired image
dimensions and applied the transformation matrix.
xtransformded T 1 x
(7.47)
To estimate the intensity of the transformed pixel xtransformded we used bilinear interpolation.
17.1.2
Histogram equalization
The histogram equalization step is optional, but sometimes it is necessary when the light
intensity difference is large.
1
H1
H2
bins
Figure 7.37 Matching between the normalized accumulative histograms of I 1 and I 2 .
We used normalized accumulative histograms H 1 and H 2 of images I 1 and I 2 . We

matched each bin from H 2 to bin in H 1 by finding the nearest value.
54
tbl
x
start point (sp)
end point (ep)
Figure 7.38 Finding the best approximation line for the histograms matched indexes.
The start point and the end point we choose according the flat areas, because flat areas do
not give us valid information about histogram equalization.
After that by using Least Square Sum we are finding the best approximation line, described
by:
y ax b
(7.48)
Where:
ep
(x
x)(tbl i tbl )
i sp
ep
(x
x) 2
i sp
(7.49)
b tbl a x
And:
x
sp ep
2
(7.50)
ep
tbl
tbl
i sp
ep sp
We equalize the intensity of I 2 according to the equation:
55
I (new
x , y ) a ( I ( x , y ) I min ) b( I max I min ) I min
(7.51)
Where I ( x , y ) is the intensity of the pixel with coordinates (x,y), I min and I max are values
of the pixels with minimum and maximum intensity, respectively.
The histogram equalization of the RGB images was applied for each color channel
separately.
17.1.3
Mosaic update
For the mosaic updating we use part of the existing image and part of the new added image
as it is shown on the Fig.(6.2).
existing
overlap
new
Figure 7.39 Mosaic updating rule.
To estimate the overlapping part we assign distance map between the centers of the images.
By this way after the back ground projection we can minimize the distortion effects at the
edges.
56
17.1.4
Algorithm
Initialize mosaic and settings
Load image
Project image
Co-register image
(optional - not implemented)
Histogram equalization (optional)
Mosaic update
Mosaic display
Figure 7.40 Mosaic construction algorithm.
As input in our algorithm we are using image sequence presented as floating point TIFF file
format images. Also we can adjust different parameters stored in a settings file.
17.2
Experiments and discussion
For the experimental part of our work we used four sets of images. Images were taken from
two different landmine fields by infrared camera and visible specter camera.
57
Figure 7.41 Mosaic constructed of the images taken from the field AccField with infrared camera and
histogram equalization during the mosaic construction.
58
Figure 7.8 Mosaic constructed with the images taken from the field AccField with regular camera,
without histogram equalization.
59
Figure 7.42 Mosaic constructed with the images taken from the field Field2230p1 with infrared camera
and histogram equalization during the mosaic construction.
60
Figure 7.43 Mosaic constructed with the images taken from the field Field2230p1 with regular camera,
without histogram equalization.
During the histogram equalization of the color images we faced some problem equalizing the
red, green and blue channel. Problems are connected with under- and overflow. Handling this
problem is connected with modification of histogram equalization algorithm, which affects
the mosaic in the far end. In a future work histogram equalization of the color can be handled
by constructing mosaic for each color channel and adjusting colors when the mosaic is
constructed completely.
61
Figure 7.44 Overlapping of the new added images with the mosaic with the infrared camera pictures.
Observing the mosaic without histogram equalization Fig. (7.11) we can easily notice how
the new images were added to the mosaic.
The speed up that we get from the C code is about three times faster than the Matlab code.
We need to take in account that Matlab code uses more system resources (memory) to
complete the same calculation task. Matlab code needs real time interpretation involving a lot
of checking, which in the C code are done during the compilation time.
62
18 Conclusions
In this report, we present C/C++ library with image co-registration, mosaicing and
corresponding tools for visualization and evaluation. The library consists of three image
registration methods, mosaic construction routines and interface to Matlab.
Optical flow image registration refers to find a displacement vector for every pixel, which
says how far it is moving across the image and the moving direction. Dense optic flow, based
on the 6-parameter affine motion estimator can only be applied to local regions. Therefore, it
is necessary to divide the images into blocks and to apply the algorithm on each block. At the
end the results are accumulated for the whole image. For better registration the method is
extended with multiresolution pyramid, constructed by successive Gaussian smoothing and
subsampling. The experimental results prove that optic flow gives good registration results.
Mutual information image registration is based on the maximization of the statistical
dependence. The implementation is suitable for different transformation models, such as
rotation and translation, linear conformal, affine and projective transformation. [50]
Pixel migration image registration uses an implicit geometrical similarity measure which is
invariant to intensity dissimilarities. The registration is achieved by tracking the migration of
high-gradient pixels using robust global optimization. We developed a multiresolution scheme
by creating pyramid of images with different resolutions. The experiment shows that the
initial estimation of the transformation parameters is more accurate by using multiresolution.
In the experiments we find some limitations of the method. For different image pairs we need
adjustment of the parameters for proper registration. These limitations could be discussed and
solved in the future research.
The programs have been embedded in a Matlab platform for testing and further
development.
The C code efficiency in mosaic construction is obvious. In the same time C programs are
bigger and much more complex than in Matlab, which affects the development time.
63
References
[1]
[2]
[3]
[4]
[5]
[6]
[7]
[8]
[9]
[10]
[11]
[12]
[13]
[14]
[15]
[16]
[17]
[18]
[19]
[20]
[21]
M. Black and A. Rangarajan. On the unification of line processes, outlier rejection and
robust statistics with application in early vision. International Journal of Computer
Vision, 19(1):57-91, 1996.
L. Brown, A survey of image registration techniques, ACM Computer Surveys, Vol.
24(4), pp. 325-376, 1992.
D. Capel A. Ziserman. Automated mosaicing with super-resolution zoom. In Proc. of
IEEE Conference on Computer Vision and Pattern Recognition, 1998.
Hua-mei Chen and Pramod K. Varshney, Mutual Information Based CR-MR Brain
Image Registration Using Generalized Partial Volume Joint Histogram Estimation,
IEEE Trans. Medical Imagein.
S.E. Chen L. Williams. View interpolation for image synthesis. In Proc. of ACM
SIGGRAPH, 1993.
S.E. Chen. Quicktime VR - An image based approach to virtual environment navigation.
In Proc. of ACM SIGGRAPH, 1995.
S. Coorg N. Master S. Teller. Acquisition of a large pose-mosaic dataset. In Proc. of
IEEE Conference on Computer Vision and Pattern Recognition, 1998.
J.D. Foley A.V. Dam S.K. Feiner J.F. Hughes. Computer Graphics: Principles and
Practice. Addison-Wesley, 2 edition, 1996.
P. Gill, Practical Optimization, Academic Press, 1982.
A. Goshtasby, John Wiley & Sons, 2-D and 3-D Image Registration, late 2003 or early
2004.
A. Goshtasbyand J. LeMoigne(Eds.), Image Registration, a special issue of Pattern
Recognition, Jan. 1999.6.
N. Greene. Environment mapping and other applications of world projections. IEEE
Transactions on Computer Graphics and Applications, pages 21-29, November 1986.
A. Guziec, X. Pennec, N. Ayache Medical Image Registration Using Geometric
Hashing, IEEE Computational Science and Engineering, October-December 1997 (Vol.
4, No. 4), pp. 29-41.
J. V. Hajnal, D. L. G. Hill, and D. J. Hawkes(Eds.), Medical Image Registration, CRS
Press, 2001.
M. Hansen P. Anandan K. Dana G. Wal P. Burt. Real-time scene stabilization and
mosaic construction. In Proc. of IEEE Workshop on Applications of Computer Vision,
pages 54-62, 1994.
B. Horn and B. Schunck. Determining optical flow. Artificial Intelligence, 17:185-203,
Aug. 1981.
M. Irani and P. Anandan, Robust Multi-Sensor Image Alignment. IEEE International
Conference on Computer Vision (ICCV), India, January 1998.
M. Irani and P. Anandan. All About Direct Methods, International Workshop on
Vision Algorithms, Corfu, Greece, September 1999.
M. Irani P. Anandan J. Bergen R. Kumar S. Hsu. Mosaic representations of video
sequences and their applications. Signal Processing: Image Communication, special
issue on Image and Video Semantics: Processing, Analysis, and Application, 8(4), May
1996.
M. Irani P. Anandan. Video indexing based on mosaic representations. Proceedings of
the IEEE, 86(5):905-921, May 1998.
M. Irani S. Hsu P. Anandan. Video compression using mosaic representations. Signal
Processing: Image Communication, 7:529-552, 1995.
64
[22] S.B. Kang R. Szeliski. 3-d scene data recovery using omnidirectional multibaseline
stereo. In Proc. of IEEE Conference on Computer Vision and Pattern Recognition,
pages 364-370, 1996.
[23] S.B. Kang. A survey of image-based rendering techniques. Technical Report CRL 97/4,
Digital Equipment Corp. Cambridge Research Lab, Aug 1997.
[24] J. Keller, Advanced Image Registration Techniques, Tel-Aviv University. October
2002.
[25] P. Kolonia. When more is better. Popular Photography, 58(1):30-34, Jan 1994.
[26] S. Kruger and A. Calway. Image registration using multiresolution frequency domain
correlation. In M. Nixon and J. Carter, editors, British Machine Vision Conference,
pages 316{325, Southampton, UK, Sept. 1998.
[27] J. Lengyel. The convergence of graphics and vision. Computer, IEEE Computer Society
Magazine, pages 46-53, July 1998.
[28] H. Li, B.S. Manjunath, and S.K. Mitra, A Contour Based Approach to multi-sensor
Image Registration, IEEE Trans. Image Processing, Vol. 4, pp. 320-334, March 1995.
[29] S. Mann R.W. Picard. Virtual bellows: Constructing high quality stills from video. In
Proc. of IEEE Int. Conf. on Image Processing, pages 363-367, 1994.
[30] Dr A D Marshall, Vision Systems Courseware, World Scientific, 1992.
[31] L. McMillan G. Bishop. Plenoptic modeling: An image-based rendering system. In
Proc. of ACM SIGGRAPH, pages 39-46, 1995.
[32] E. Memin and P. Perez. Dense estimation and object-based segmentation of the optical
flow with robust techniques. IEEE Transactions on Image Processing, 7(5):703-719,
1998.
[33] C. H. Morimoto R. Chellappa. Fast 3d stabilization and mosaic construction. In Proc. of
IEEE Conference on Computer Vision and Pattern Recognition, pages 660-665, June
1997.
[34] M. Obitko, Marek and P. Slavk, "Visualization of Genetic Algorithms in a Learning
Environment", Spring Conference on Computer Graphics, SCCG'99. Bratislava,
Comenius University, 1999, p. 101-106. ISBN 80-223-1357-2
[35] Josien Pluim, J. B. Maintz and M.Viergever. Image Registration by Maximization of
Combined Mutual Information and Gradient Information, IEEE Transactions on
Medical Imagining, Vol. 19, NO. 8, August 2000.
[36] I. Pratikakis, C. Barillot, P. Hellier, and E. Memin. Robust multiscale deformable
registration of 3D ultrasound images. International Journal of Image and Graphics,
3(4):1-19, 2003.
[37] K. Rohr, Landmark-Based Image Analysis: Using Geometric and Intensity Models,
KluwerAcademic Publishers, Boston, MA, 2001.
[38] H.S. Sawhney S. Ayer. Compact representation of videos through dominant multiple
motion estimation. IEEE Transactions on Pattern Analysis and Machine Intelligence,
18(8):814-830, 1996.
[39] R. Sharma andM. Pavel, Registration of video sequences frommultiple sensors in
Proceedings of Image Registration Workshop, Publication CP-1998-206853, NASAGSFC, November 1997.
[40] H.Y. Shum M. Han R. Szeliski. Interactive construction of 3d models from panoramic
mosaics. In Proc. of IEEE Conference on Computer Vision and Pattern Recognition,
June 1998.
[41] M. Sonkaand J. M. Fitzpatrick (Eds.), Handbook of Medical Imaging, vol. 2, Image
Registration,SPIE Press, 447506, 2000.
[42] R. Szeliski H.Y. Shum. Creating full view panoramic image mosaics and environment
maps. In Proc. of ACM SIGGRAPH, 1997.
65
[43] R. Szeliski. Image mosaicing for tele-reality applications. In Proc. of IEEE Workshop
on Applications of Computer Vision, pages 44-53, 1994.
[44] R. Szeliski. Video mosaics for virtual environments. IEEE Computer Graphics and
Applications, pages 22-30, March 1996.
[45] M. A. Unser, A. Aldroubi, and C. R. Gerfen. Multiresolution image registration
procedure using spline pyramids. In A. F. Laine, editor, Proc. SPIE Vol. 2034, Wavelet
Applications in Signal and Image Processing, pages 160-170, Orlando (FL), USA, Apr.
1993.
[46] A. Watt. 3D Computer Graphics. Addison-Wesley, 2 edition, 1995.
[47] W.M.Wells III, P. Viola, H. Atsumi, S. Nakajima, and R. Kikinis, Multi-modal volume
registration by maximization of mutual information, Med. Image Anal., Vol. 1, no. 1,
pp. 35-51, 1996.
[48] P.R. Wolf. Elements of Photogrammetry. McGraw-Hill, 2 edition, 1983.
[49] Y. Xiong K. Turkowski. Creating image-based VR using a self-calibrating fisheye lens.
In Proc. of IEEE Conference on Computer Vision and Pattern Recognition, 1997.
[50] Q. Zhang, Image Registration Based on Mutual Information, department Electronics
option Information and Communication Technologies, Technologische Hogeschool
Leuven.
[51] SMILE. European workshop on 3d structure from multiple images of large-scale
environments (in conjunction with eccv), 1998.
[52] Sarnoff Corporation, Mosaic Construction,
http://www.sarnoff.com/products_services/government_solutions/vision_technology/cor
e_technology/mosaic_construction.asp
66

BSC Thesis Todor Balabanov NBU 2006

Hochgeladen von

Dokumentinformationen

Originaltitel

Copyright

Verfügbare Formate

Dieses Dokument teilen

Dokument teilen oder einbetten

Freigabeoptionen

Stufen Sie dieses Dokument als nützlich ein?

Sind diese Inhalte unangemessen?

Copyright:

Verfügbare Formate

BSC Thesis Todor Balabanov NBU 2006

Hochgeladen von

Copyright:

Verfügbare Formate

NEW BULGARIAN UNIVERSITY

BACHELOR PROGRAM INFORMATICS

Library for image co-registration with

Dense optic flow....................................................................................................... 19

Selection of registration methods and parameters.................................................... 46

Limitations of user interface..................................................................................... 49

Why we need mosaic construction in ClearFast project?......................................51

1.2 Structure of the report

Figure 2.3 Optic flow code structure.

2.2 Calling conventions

The matrix is with wrong dimensions.

The library consists of four common functions:

3.1 Data types/structure

3.2 Functions list

Calculate 'vector' norm. Size of the vector is 'size'.

3.3 Memory allocations

4 Optical flow based registration

5 Dense optic flow

where s and sr are auxiliary variables acting as weights.

dwsk wr^ k dwrk

At each level of resolution, a multigrid minimization based on successive partitions of the

5.2 Experiments and discussion

Figure 3.8 Taxi sequence images. a)Fist frame. b)Second frame.

6 Pixel migration based registration

Where S1 is a set of pixels in I1 and || . || is a Euclidean norm.

while the set S1(p) is given by:

where p Fn is the gradient vector of F with respect of p at iteration n, H pn is the Hessian

where shortly we denote XAffine or XProjective as X:

gets bigger than in the previous iteration.

Figure 4.11 Multiresolution pyramid is used to estimate initial values for p.

pn+1 = pn + Inv(Hpn)* pFn

Figure 4.13 Diagram for Newtons iterative optimization.

genetic algorithms parameters. The convergence can be controlled by parameters as follows:

Figure 4.14 Diagram for genetic algorithm optimization.

9.2 Experiments and discussion

Figure 4.16 Lenas transformed image, with Newtons iterative optimization.

Figure 4.18 Lenas transformed image, with genetic algorithm.

10 Mutual information based registration

The MI of two random variables A and B is defined by:

Nonlinear conjugate gradient method

Consider the unconstrained optimization problem

where g k f ( x k ) , k is a stepsize obtained by a one-dimensional line search and k is

respectively, where means the Euclidean norm and y k 1 g k g k 1 . If the condition

need not converge.

Figure 5.28 Flow chart of the MI based registration system.

Experiments and discussion

Figure 5.29 Agricultural area pictures. Left 09/07/94. Right 21/09/92.

Figure 5.30 Agricultural area transformed image.

Figure 6.31 User interface main working screen.

Selection of registration methods and parameters

Figure 6.32 Optic Flow Options Dialog.

Figure 6.33 Mutual Information Options Dialog.

Figure 6.34 Pixel migration options dialog.

Limitations of user interface

Why we need mosaic construction in ClearFast project?

Theory and formulations

Figure 7.35 Camera model for the ClearFast project.

1. Image back-projection onto ground surface;

p1t p 2t tan(elr ) p3t tan(azr )

p1t p 2t tan(elr ) p3t tan(azr )

And t is a tilt angle, M is transformation matrix correcting the tilt.

Where el is elevation angle and az is azimuth angle.

Where R is the distance of the camera to the point on the ground.