Beruflich Dokumente
Kultur Dokumente
Outline
A. One camera
Lm
M1
M'' M'
m
B. Two cameras
extrinsic parameters
M m1 m2
C2
A. One camera
Lm
M1
M'' M'
m
Perspective
visual sensors form images by concentrating light on a sensitive surface using lenses Images are a projective mapping of the 3D space understanding this mapping is essential for inverting it
Perspective (cont'd)
Intuitively, we have a relation between apparent size and depth of the object
f x= X d
u m@ v A w
w=0
point at infinity
m1 m2
l1
line at infinity
l2
ax + by + cw = l m = 0
x y a +b +c =0 w w
a l @ b A c
m
l3
l1 l2
m1 m2 m3
x B y C C M B @ z A w
M1
M L
h O
nx B ny C C B @ nz A h
M = 0; 8M 2
= m
L O m
L (; )
x x B B y C C homogeneous B @ y A B @ @ z A z 1 m m M
to
Euclidian
0 1 x x C 1 @ y C y A z A w z w M
from
Euclidian
Projective geometry
points at infinity (equivalent to Euclidian vectors): e.g. parallel lines meet at infinity allows for a linear representation of translations:
t = (tx ; ty ; tz )>
0 1 0 e.g., rotation 0 1 0 0 0 0 1 0
translation
x + tx 1 B y + ty C B 0 B C=B @ z + tz A @ 0 1 0
tx x ty C B y C CB C tz A @ z A 1 1
1 0
Projection matrix
Using homogeneous coordinates, the pinhole model can then be formalized as: where
m= AM
f A=@ 0 0
0 f 0
0 0 1
0 0 A 0
s is the skew (generally null) pixel arrangement u and v are the pixel resolution in u and v c = (u0 ; v0 )> is the projection of the optical center.
m= K AM 0 1 u s u 0 K = @ 0 v v0 A 0 0 1
Calibration
f u 0 KA=@ 0
f s f v 0
u0 v0 1
0 0 A 0
s f u ; f v ; ; u0 ; v0 u
Distortion
Real lenses are only an approximation of a pinhole model => distortion e.g. radial distortion with wide-angle (small focal length) lenses. Lead to non-linear projection model
Calibration (2)
openCV
http://www.intel.com/technology/computing/opencv/
Based on regular grid patterns of known sizes, with manual or automatic corner extraction provides an estimation of all parameters + distortion provides tool for image rectification
Calibration(3)
B. Two cameras
M m1 m2
C1
C2
3D Pose
translation between the two optical centers rotation between optical axes
m1
C1
m2
C2
Extrinsic parameters
m = (K A H) M
H=
R 013
t 1
R is the rotation that brings the optical axis to the z -axis t is the translation vector that brings C onto the origin
O = (0; 0; 0; 1)>
t = R C
Stereo Calibration
http://www.vision.caltech.edu/bouguetj/calib_doc/htmls/example5.html
Practical
Objectives:
compute the projection matrices P1 and P2 compute the fundamental matrix F12 display both left and right images when a point is clicked in the left image, displays the corresponding epipolar line in the right image reconstruct the corresponding 3D point when a stereo pair of points is selected.
Outline
A. One camera
Lm
M1
M'' M'
B. Two cameras
extrinsic parameters
M m1 m2
C2
L O
~ c ~ a
.
~ b
Back:
Epipolar geometry
Plausible 3D origins for an image point m form a halfline in space. This line passes through the optical center of the camera.
Lm
M'' M'
M1
Here italic denotes to projective quantities, e.g., P and bold denotes Euclidean quantities, e.g., P P is a 3x3 matrix. p is a dimension 3 column vector.
Note that p contains the projective coordinates of the projection on the image plane of the world coordinate system's origin.
C = R 1 t = P 1 p
M1 =
P 1 m 0
Back-Projected ray
From this we can compute the coordinates of a line in space where lie all plausible 3D origins of the image point
Lm =
1 kM
1k
(M
1; C
1)
Effectively we can even say that the plausible origins lie on the half line (or ray) starting at the camera's optical center C This limits the possible locations of the correspondence in the second image to a line, a half line, or even a segment depending on the set-up.
Epipolar line
Xl X l
Lxl Il
Ir xl
xr
x r
Cl
er
Cr
The plausible correspondences on the right image form a 2Dline that is the re-projection of the optical ray on the image It can be a line, a half-line or even a segment, depending if the endpoints lie at infinity or not. One endpoint is fixed and is e = P C 2 2 1 called the Epipole Note that if the optical center lie behind the camera, the epipole is not a boundary The second is the reprojection at infinity of the point considered:
e2
C2 e1
m2 m1
m2;1 = P2 M1;1
C1
Then the line between those two points, where lie all plausible correspondences, is called the Epipolar line
(a)
(b)
l = e2 m2;1
(c)
(d)
e2
e2
e2
e2
C1 C1 C2 C1 C2 C2
C2 C1
Example (stereo)
C1
LEFT STEREO
C2
RIGHT
> 2
F12 m
=0
for m 1 and m 2 two corresponding points, respectively in the left and right image.
0 [v] = @ v3 v2
v3 0 v1
v2 v1 A 0
P P+ P = P
P + = H 1 A+ K 1
l = F12 m
the distance from the correspondence to the epipolar line gives one constraint m> F12 m1 = 0 2 Solve a system of linear equations (RANSAC will be discussed in the Motion Estimation lecture)
Feature correspondences
left
right
correspondences
left
right
left camera
right camera
Point reconstruction
Two corresponding stereo points back-project two optical rays in space. Theoretically, M lie at the intersection between the two lines. Practically, the two lines are unlikely to intersect. Often the closest point to both lines is chosen.
a)
m1
m2
C1
C2
b)
L2 M1
L1 M2
1 (2 2 ) (1 2 ) 1 (2 1 ) = 1 + 1 1 2 jj1 2 jj
a)
m1
m2
L1 = (1 ; 1 ) L2 = (2 ; 2 ) k1 k = k2 k = 1
C2
b)
L2 M1
L1 M2
1 (2 2 ) (1 2 ) 1 (2 1 ) = 1 + 1 1 2 jj1 2 jj
Practical
Objectives:
compute the projection matrices P1 and P2 compute the fundamental matrix F12 display both left and right images when a point is clicked in the left image, displays the corresponding epipolar line in the right image reconstruct the corresponding 3D point when a stereo pair of points is selected.
Linear alternative
We can alternatively formulate the reconstruction as a linear optimization problem: Al bl M = Ar br Namely if we rewrite the projection matrices as: P = @
0 Q> 1 Q> 2 Q> 3
Then we can write the projection as the system of linear equations: 0 1 0 1 > Q1 q14 u M @ Q> q24 A = @ v A 2 1 > 1 Q3 q34
Linear formulation
Which translates into the system of equations 8 > < Q1 M + q14 = u Q> M + q24 = v : 2 Q> M + q34 = 3 By replacing we have:
Q> M uQ> M + q14 uq34 1 3 Q> M vQ> M + q24 vq34 2 3 > > Q1 uQ3 And when reordering: M = > > Q2 vQ3
= =
0 0
Ai
bi
We concatenate the equations provided by the corresponding points in both images into one system:
A M = b with
= =
Al Ar bl br
Line reconstruction
The plausible origins of 2D lines form a plane in space The intersection of the two planes generated from the line in both images give a reconstructed 3D-line This process is ill-defined if the lines are parallel to the epipolar line -> then the two lines back-project the same plane in 3D !
L = (1 2 ; h2 1 h1 2 )
Image rectification
Make epipolar lines horizontal Note: there is still an horizontal shift of the images !
Points infinitely far do not have zero disparity
e2
rectification
e2
C1
C2
C1
C2
after
left
right