Beruflich Dokumente
Kultur Dokumente
1
Epipolar geometry
Thomas Opsahl
Weekly overview – Two-view geometry
• Epipolar geometry
– Algebraic representation & estimation
– The essential matrix
– The fundamental matrix
• Triangulation
– Sparse 3D scene reconstructing from 2D
correspondences
2
Introduction
• Observing the same points in two views puts a 𝑿
strong geometrical constraint on the cameras
𝒙 𝒙𝒙
• Algebraically this epipolar constraint is usually
represented by two related 3 × 3 matrices
𝐶 𝐶′
𝒖 𝒖𝒖
3
Introduction
• Observing the same points in two views puts a 𝑿
strong geometrical constraint on the cameras
𝒙 𝒙𝒙
• Algebraically this epipolar constraint is usually
𝐸
represented by two related 3 × 3 matrices
𝐶 𝐶′
• The fundamental matrix 𝐹
� ′𝑇 𝐹𝒖
𝒖 � 𝐾 𝐾𝐾
�′𝑇 𝐸𝒙
𝒙 �
4
The essential matrix E
• Let 𝐶𝒙 ↔ 𝐶𝐶𝒙′ be corresponding points in the
normalized image planes and let the pose of 𝐶
relative to 𝐶𝐶 be
𝐶𝐶 𝑅 𝒕
𝜉𝐶 =
𝟎 1
𝐶𝒙
𝐶′𝒙′
𝐶
𝒕
𝐶′
5
The essential matrix E
• Let 𝐶𝒙 ↔ 𝐶𝐶𝒙′ be corresponding points in the
normalized image planes and let the pose of 𝐶
relative to 𝐶𝐶 be
𝐶′ 𝑅 𝒕
𝜉𝐶 =
𝟎 1
�′𝑇 𝐸𝒙
𝒙 �=0
6
The essential matrix E
• �′𝑇 𝐸𝒙
Note that although 𝒙 � = 0 is a necessary
requirement for the correspondence 𝒙 ↔ 𝒙𝒙 to
be geometrically possible, it does not guarantee
its correctness 𝐶𝒙
𝐶′𝒙′
𝐶𝒙
�
𝐶′𝒙
�′
𝐶
𝒕
𝐶′
8
The essential matrix E
𝑿
• Properties of 𝐸
– 𝐸 = 𝒕 ×𝑅
𝒙 𝒙𝒙
– Homogeneous
𝐸
– 𝑟𝑟𝑟𝑟 = 2
𝐶 𝐶′
– det = 0
– 5 degrees of freedom
– 𝐸 can be estimated from a minimum of 5
point correspondences
– If 𝒙 and 𝒙𝒙 are corresponding normalized • It is possible to decompose 𝐸 = 𝒕 × 𝑅 to
image points, then 𝒙 �′𝑇 𝐸𝒙
�=0 determine the relative pose between
– 𝐸 has 2 singular values that are equal and cameras
a third that is zero – Translation only up to scale
– Topic of another lecture
9
The fundamental matrix F
𝑿
• The epipolar constraint on image points is naturally
connected to the essential matrix by the calibration
matrices 𝐾 and 𝐾′ 𝒙 𝒙𝒙
K C x =u ⇒ C x =K −1u 𝐸
C′ C′ C′
𝐶 𝐶′
K ′ x ′ =u ′ ⇒ x ′ =K ′ u ′ ⇒ x ′ =
−1
u ′ K ′
T T −T
u ′T K ′−T EK −1u = 0
� ′𝑇 𝐹𝒖
𝒖 �=0
10
The fundamental matrix F
• Properties of 𝐹 𝑿
−𝑇 −1
– 𝐹 = 𝐾𝐾 𝐸𝐾
– Homogeneous
– 𝑟𝑟𝑟𝑟 = 2 𝒙 𝒙𝒙
– det = 0
– 7 degrees of freedom 𝐶 𝐶′
– 𝐹 can be estimated from a minimum of 7 point
correspondences
𝐾 𝐾𝐾
– If 𝒖 and 𝒖𝒖 are corresponding image points, then
� ′𝑇 𝐹𝒖
𝒖 �=0
– For any point 𝒖 in image 1, the corresponding 𝒖 𝐹 𝒖𝒖
epipolar line 𝒍′ in image 2 is given by
𝒍̃′ = 𝐹𝒖�
– The epipole 𝒆𝒆 in image 2 is 𝐹’s left singular
– For any point 𝒖𝒖 in image 2, the corresponding vector corresponding to the zero singular value
epipolar line 𝒍 in image 1 is given by – The epipole 𝒆 in image 1 is 𝐹’s right singular
𝒍̃ = 𝐹 𝑇 𝒖
�′ vector corresponding to the zero singular value
11
Example
𝒖𝒖
• These fundamental lines were determined using the fundamental matrix between images
• Recall that points and lines are dual in ℙ2
u
l T u = 0 ⇔ [l0 l1 l2 ] v = 0 ⇔ l0u + l1v + l2 = 0
1
12
Estimating F
13
Estimating F
𝑿
The normalized 8-point algorithm
Given 𝑛 ≥ 8 correspondences 𝒖𝑖 ↔ 𝒖𝑖 ′, do the
following 𝒙 𝒙𝒙
𝐶 𝐶′
1. Normalize points 𝒖𝑖 and 𝒖𝑖 ′ usingsimilarity
transforms 𝑇 and 𝑇𝑇
2. Build matrix 𝐴 from point-correspondences and 𝐾 𝐾𝐾
compute its SVD
3. Extract the “solution” 𝐹� from the right singular 𝒖 𝐹 𝒖𝒖
vector corresponding to the smallest singular
value
4. Compute the SVD of 𝐹� : 𝐹� = 𝑈𝑈𝑉 𝑇 𝑇 𝑇𝑇
5. Enforce zero determinant by setting 𝑠33 = 0 and
compute a proper fundamental matrix �
𝒖
𝐹�
�′
𝒖
𝐹� = 𝑈𝑈𝑉 𝑇
uˆ ′T Fˆuˆ = 0
6. Denormalize 𝐹 = 𝑇𝑇𝑇 𝐹� 𝑇
(T ′u′ ) Fˆ (Tu ) =0 ⇒ F =T ′T FT
T
ˆ
u′T T ′T FT
ˆ u=0 14
Estimating F
The normalized 8-point algorithm The 7-point algorithm
Given 𝑛 ≥ 8 correspondences 𝒖𝑖 ↔ 𝒖𝑖 ′, do the • Given 7 correspondences, 𝐴 will be a 7 × 9 matrix
following which in general will be of rank 7
1. Normalize points 𝒖𝑖 and 𝒖𝑖 ′ usingsimilarity • So the null space of 𝐴 is 2-dimensional and the
fundamental matrix must be a linear combination of
transforms 𝑇 and 𝑇𝑇 the two right null vectors of 𝐴
2. Build matrix 𝐴 from point-correspondences and 𝒇 𝛼 = 𝛼𝒇1 + 1 − 𝛼 𝒇2
compute its SVD 𝐹 𝛼 = 𝛼𝐹1 + 1 − 𝛼 𝐹2
3. Extract the “solution” 𝐹� from the right singular
vector corresponding to the smallest singular • The additional constraint 𝑑𝑑𝑑 𝐹 = 0 leads to a
value cubic polynomial equation in 𝛼 which has 1 or 3
4. Compute the SVD of 𝐹� : 𝐹� = 𝑈𝑈𝑉 𝑇 solutions 𝛼𝑖 which in turn yields 1 or 3 𝐹’s
5. Enforce zero determinant by setting 𝑠33 = 0 and
compute a proper fundamental matrix • This algorithm is to prefer in a RANSAC scheme,
since it is minimal and since for a single sampling
𝐹� = 𝑈𝑈𝑉 𝑇 of 7 correspondences one might get 3 fundamental
6. Denormalize 𝐹 = 𝑇𝑇𝑇 𝐹� 𝑇 matrices to test for inliers
15
Estimating F
• Improved estimates of 𝐹 can be obtained using • OpenCV
iterative methods like Levenberg-Marquardt to – cv::Mat cv::findFundamentalMat
minimize the epipolar error – Arguments are
InputArray points1
�, 𝐹𝑇 𝒖
𝜖 = �𝑑 𝒖 �′ + 𝑑 𝒖
� ′, 𝐹𝒖
�
InputArray points2
int method – {CV_FM_7POINT, CV_FM_8POINT,
𝒖 𝒖𝒖 CV_FM_RANSAC, CV_FM_LMEDS}
𝐹 double param1
double param2
OutputArray mask
𝒍̃ = 𝐹 𝑇 𝒖
�′ 𝒍̃′ = 𝐹𝒖
�
• Matlab
– estimateFundamentalMatrix
16
Estimating F
• Improved estimates of 𝐹 can be obtained using • OpenCV
iterative methods like Levenberg-Marquardt to – cv::Mat cv::findFundamentalMat
minimize the epipolar error – Arguments are
InputArray points1
�, 𝐹𝑇 𝒖
𝜖 = �𝑑 𝒖 �′ + 𝑑 𝒖
� ′, 𝐹𝒖
�
InputArray points2
int method – {CV_FM_7POINT, CV_FM_8POINT,
𝒖 𝒖𝒖 CV_FM_RANSAC, CV_FM_LMEDS}
�, 𝐹𝑇 𝒖
𝑑 𝒖 �′ 𝐹 double param1
� ′, 𝐹𝒖
𝑑 𝒖 � double param2
OutputArray mask
𝒍̃ = 𝐹 𝑇 𝒖
�′ 𝒍̃′ = 𝐹𝒖
�
• Matlab
– estimateFundamentalMatrix
• � and line
Distance between homogeneous point 𝒖
𝑻
𝒍̃ = 𝑙̃1 , 𝑙̃2 , 𝑙̃3
T
( )
d u , l =
u l
l 2 + l 2
1 2
17
Estimating E
• OpenGV: http://laurentkneip.github.io/opengv/
• In RANSAC schemes, the 5-point algorithm is
contains several interesting functions
the fastest alternative
– 5-pt algorithm
– To acieve 99% confidence with 50% outliers, the
5-point algorithm only requires 145 tests while – 2-pt algorithm based on known relative rotation
the 8-point algorithm requires 1177 tests
18
Summary
𝒖 𝐹 𝒖𝒖
• Additional reading:
– Szeliski: 7.2
19