Trifocal tensors Euclideanprojective SFM Self calibration Line geometry Purely projective cameras Je ne suis pas la la semaine prochaine Quand peut on rattrapper le cours ID: 374124
Download Presentation The PPT/PDF document "Projective cameras" is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
Projective cameras
Trifocal tensorsEuclidean/projective SFMSelf calibrationLine geometryPurely projective cameras
Je ne
suis
pas la la
semaine
prochaine
.
Quand
peut
-on
rattrapper
le
cours
?
Planches
:
http://www.di.ens.fr/~
ponce/geomvis/lect5.pptx
http://www.di.ens.fr/~ponce/geomvis/lect5.pdfSlide2
Trinocular Epipolar Constraints
These constraints are not independent!Slide3
Trifocal ConstraintsSlide4
Trifocal Constraints
All 3x3 minors
must be zero!
Calibrated CaseSlide5
Trifocal Constraints
All 3x3 minors
must be zero!
Calibrated Case
Trifocal TensorSlide6
Trifocal Constraints
All 3x3 minors
must be zero!
Calibrated Case
Trifocal TensorSlide7
Trifocal Constraints
All 3x3 minors
must be zero!
Calibrated Case
Trifocal TensorSlide8
Trifocal Constraints
Uncalibrated Case
Trifocal TensorSlide9
Trifocal Constraints: 3 Points
Pick any two lines
l
and
l
through
p
and
p
.
Do it again.
2
3
2
3
T(
p , p , p
)=0
1
2
3Slide10
Properties of the Trifocal Tensor
Estimating the Trifocal Tensor Ignore the non-linear constraints and use linear least-
squares a
posteriori.
Impose the constraints a posteriori.
For any matching
epipolar
lines,
l G l = 0.
The matrices G are singular. They satisfy 8 independent constraints in the
uncalibrated
case (
Faugeras
and
Mourrain
, 1995).
2
1
3
T
i
1
iSlide11
For
any matching epipolar lines, l G l = 0.
2
1
3
T
i
The backprojections of the two lines do not define a line!Slide12
Multiple Views (Faugeras and Mourrain, 1995)Slide13
Two Views
Epipolar ConstraintSlide14
Three Views
Trifocal ConstraintSlide15
Four Views
Quadrifocal Constraint
(Triggs, 1995)Slide16
The
Euclidean (perspective) Structure-from-Motion ProblemGiven m calibrated perspective images of n fixed points Pj we can write
Problem:
estimate the
m
3x4 matrices
Mi = [Ri t
i] andthe n positions Pj from the mn correspondences pij .
2mn equations in 11m
+3
n
unknowns
Overconstrained problem, that can be solved
using (non-linear) least squares!Slide17
The Euclidean Ambiguity of Euclidean SFM
If Ri, ti, and Pj are solutions,
So are
R
i
’,
t
i’, and Pj’, where
In fact, the absolute scale cannot be recovered since:When the intrinsic and extrinsic parameters are known
Euclidean ambiguity up to a similarity
transformation. Slide18
Euclidean
motion from E (Longuet-Higgins, 1981)Given F computed from n > 7 point correspondences, and its SVD F= UWVT, compute E=U diag(1,1,0) VT.
There are two solutions t’ = u
3
and t’’ = -t’ to
E
T
t=0.Define R’ = UWV
T and R” = UWTVT where(It is easy to check R’ and R” are rotations.) Then [
tx’]R’ = -E and [tx’]R” = E. Similar reasoning for t”.
Four solutions. Only two of them place the reconstructed
points in front of the cameras.Slide19
Euclidean reconstruction. Mean relative error: 3.1%Slide20
A different view of the fundamental matrix
Projective ambiguity ! M’Q=[Id 0] MQ=[A b]. Hence: zp = [A b] P and z’p’ = [Id 0] P, with P=(x,y,z,1)T. This can be rewritten as:
zp
= ( A [Id 0] + [0 b] ) P =
z’Ap
’ + b.
Or: z (b
× p) = z’ (b × Ap
’). Finally: pTFp’ = 0 with F = [bx] A.Slide21
Projective motion from the fundamental matrix
Given F computed from n > 7 point correspondences, compute b as the solution of FTb=0 with |b|2=1.Note that: [ax]
2
= aa
T
- |a|
2
Id for any a. Thus, if A0 = - [b
x] F, [bx] A0 = - [bx]
2 F = - bbTF + |b|2
F = F.
The general solution is M = [A b] with
A =
A
0
+ (
b |
b | b).Slide22
Two-view projective reconstruction. Mean relative error: 3.0%Slide23
Bundle adjustment
Use nonlinear least-squares to minimize:Slide24
Bundle adjustment. Mean relative error: 0.2%Slide25
From
uncalibrated to calibrated cameras
Weak-perspective camera:
Calibrated camera:
Problem: what is
Q
?
Note:
Absolute scale cannot be recovered. The
Euclidean shape
(defined up to an arbitrary similitude) is recovered.Slide26
Reconstruction Results (
Tomasi and Kanade, 1992)
Reprinted from “Factoring Image Sequences into Shape and Motion,” by C. Tomasi and
T. Kanade, Proc. IEEE Workshop on Visual Motion (1991).
1991 IEEE.Slide27
What is
some parameters are known? Self calibrationM= ρ K [ R t ]
Q= [C d] => M (CC ) M
=
ρ
KK
T
T
T
2
M
Ω
M
=
ρ
ω
with
Ω
= CC and
ω
= KK
T
T
T
*
*
*
*
If
u0=v0=0
, linear constraints on
Ω
(Triggs’97, Pollefeys’98)
*
2Slide28
x
c
ξ
r
y
c
Purely projective camerasSlide29
x
c
ξ
r
y
x
c
ξ
Line geometry! Slide30
x
y
=
x
Ç y = = = (u
; v)
y – x u
x
£
y v
The join of two points and
Plücker
coordinates
(Euclidean
version)
u
O
v
Note:
u . v
= 0Slide31
x
y
The join of two points and Plücker coordinates
(projective version)
u
O
v
Note:
u . v
= 0
=
x
Ç
y
=
u
v
[ ]Slide32
An inner product for lines
= (u ; v) ! * = (v ; u)
= (
s ; t
)
!
( |
) = * . =
. * = u . t + v . s
(
|
) = 0
Note:
(
|
)
= 2 u . v = 0Slide33
line
screw
P
5
the Klein quadric
Interpreting Pl
ű
cker
coordinatesSlide34
p
2Duality
x
p
1
p
3
x . p
k
= 0
x
*
= {
p | x . p
= 0 }Slide35
= p Æ q = ( p Ç q )*
The meet of two planes
p
q
Slide36
* = x* Æ y* = ( x Ç y )
*
Line duality
x
*
y
*
*
x
ySlide37
The joint of a line and a point
x
p
=
Ç x
p = [ Ç] x where [ Ç] =
[u
£
]
v
-v
T
0
When
is
Ç x
equal to 0?Slide38
The meet of a line and a plane
p
x
=
Æ p
x = [Æ] p where [Æ] = [
*Ç]
When if
Æ
p
equal to
0?Slide39
Coplanar lines
and
Line bundlesSlide40
»
= u1 »1 + u2 »
2
+
u
3
»3
Line bundlesc
x
x
3
x
1
x
2
xSlide41
»
= u1 »1 + u2 »
2
+
u
3
»3 y =
u1 y1 + u2
y2 + u3
y
3
y
2
c
r
x
y
y
1
x
3
x
1
x
2
x
y
3
Line bundlesSlide42
»
= X u , where X2R6£3, u2
R
3
y
=
Y u ,
where Y2R
4£3, u2R3
y
2
c
r
x
y
y
1
x
3
x
1
x
2
y
3
Line bundles
xSlide43
u
= Yzy y = Yzy [(c Ç
x
)
Æ
r
]
y
2
c
r
x
y
y
1
x
3
x
1
x
2
y
3
Line bundles
x
Note:Slide44
u
= Yzy y = Yzy [(c Ç
x
)
Æ
r
]
y
2
c
r
x
y
y
1
x
3
x
1
x
2
y
3
Line bundles
x
Note:
(
c
Ç
x
)
Æ
r =
[
c x – x c
]
r
T
TSlide45
u
= Yzy y = Yzy [(c Ç
x
)
Æ
r
] = P x when
z = c
y
2
c
r
x
y
y
1
x
3
x
1
x
2
y
3
Line bundles
xSlide46
c
r
x
y
x
y
¼
P x
m
¼
P
*
l
p
¼
P
T
m
¼ P T
y
Perspective projection
z
l
m
p
NOTE:
Here
u=y
X=P
TSlide47
The fundamental matrix revisited
(»
1
|
»
2) = 0
y1T F y2 = 0
y11
2
1
¼
P
1
T
y
12 ¼ P2
T y2
y
2Slide48
1
1
1
2
2
2
3
3
34
4 4
5
5 5
6 6
6
The trifocaltensor
revisited
T (
y
1
,
y
2
,
y
3
) = 0Slide49
The trifocal tensor revisited
Di (»1 , »2
,
»
3
) = 0 or
Ti (u1 , u2 , u
3 ) = 0, for
i
=
1,2,3,4
1
1 1
2
2 2
3
3
3
4
4
45
5
5
6
6
6
δ
η
φ
x
(Ponce et al., CVPR’05)Slide50
c
r
x
y
x
y
¼
P x
m
¼
P
*
l
p
¼
P
T
m
¼ P T y
Perspective projection
z
l
m
p
NOTE:
Here
u=y
X=P
TSlide51
П
1
Chasles’ absolute conic:
x
1
2
+
x
2
2
+
x
3
2
= 0,
x
4
= 0.
The absolute quadratic complex:
T diag(Id,0) d = | u
|2 = 0.Slide52
e
pT = H ppTex = H-1
p
x
Coordinate changes --- Metric upgrades
Planes:
Points:
Lines:
e
= p
x’
¼
P
(
H H
-1
)
x
H =
[
X y ]Slide53
Perspective projection
c
r
x
x
’
x
c
r
x
x
’
x
x
’
¼
P x
d
’
¼
P
*
d
p
’
¼
P
T
d
’
x
¼
P
T
x’
x
’
¼
P x
d
¼
P
*
d
p
’
¼
P
T
d
x
¼
P
T
The AQC general equation:
d
T
= 0, with
=
X
*T
X
*
Proposition:
T
’
¼
û
¢
û
’
Proposition :
P
P
T
¼
’
p
d
y
’
d
’
y
’
d
’
Proposition :
P
*
P
T
¼
*
Triggs (1997);
Pollefeys et al. (1998)
e
p
T
= H
p
p
T
e
x = H
-1
p
x
e
=
p
Slide54
Relation between
K, , and *Slide55
2480 points tracked in 196 images
Non-linear, 7 imagesNon-linear, 20 imagesNon-linear, 196 images
Linear, 20 imagesSlide56
Canon XL1 digital camcorder, 480
£720 pixel2 (Ponce & McHenry, 2004)
Projective structure from motion : Mahamud, Hebert, Omori & Ponce (2001)Slide57
What is a camera?
(Ponce, CVPR’09)
x
c
ξ
r
y
xSlide58
x
c
ξ
r
y
cSlide59
x
c
ξ
r
y
x
c
ξSlide60
x
c
ξ
r
y
x
c
ξ
ξSlide61
x
c
ξ
r
y
x
ξ
r
y
Linear
family
of lines
x
ξ
x
c
ξ
ξ
ξSlide62
Lines linearly dependent on 2 or 3 lines
(Veblen & Young, 1910)
Then go on recursively for general linear dependence
© H. Havlicek, VUTSlide63
What a camera is
Definition: A camera is a two-parameter linear family of lines – that is, a degenerate regulus, or a non-degenerate linear congruence.Slide64
Rank-3 families:
Reguli
Line fields
≡
epipolar plane images
(Bolles, Baker, Marimont, 1987)
Line bundlesSlide65
Rank-4 (nondegenerate) families:
Linear congruencesFigures © H. Havlicek, VUTSlide66
x
ξ
y
r
x
y
r
ξ
Hyperbolic linear congruences
Crossed-slit cameras
(Zomet et al., 2003)
Linear pushbroom cameras
(Gupta & Hartley, 1997)Slide67
© E. Molzcan
© Leica
Hyperbolic linear congruencesSlide68
© T. Pajdla, CTU
Elliptic linear congruences
Linear oblique cameras (Pajdla, 2002)
Bilinear cameras (Yu & McMillan, 2004)
Stereo panoramas / cyclographs
(Seitz & Kim, 2002)Slide69
Parabolic linear congruences
Pencil cameras (Yu & McMillam, 2004)Axial cameras (Sturm, 2005)Slide70
Plücker coordinates and the Klein quadric
line
screw
the Klein quadric
=
x
Ç
y
=
u
v
[ ]
x
y
Note:
u . v
= 0
P
5Slide71
Pencils of screws and linear congruences
line
s
P
5
the Klein quadric
Reciprocal screws:
(s | t) = 0
Screw
≈
linear complex:
s ≈ {
±
| ( s |
±
) = 0 } Slide72
line
s
P
5
the Klein quadric
t
l
Pencils of screws and linear congruences
Reciprocal screws:
(s | t) = 0
Screw
≈
linear complex:
s ≈ {
±
| ( s |
±
) = 0 }
Pencil of screws:
l = {
¸
s +
¹
t ;
¸
,
¹
2
R
}The carrier of l
is alinear congruenceSlide73
P
5
e
h
p
Reciprocal screws:
(s | t) = 0
Screw
≈
linear complex:
s ≈ {
±
| ( s |
±
) = 0 }
Pencil of screws:
l = {
¸
s +
¹
t ;
¸
,
¹
2
R
}
The carrier of
l
is a
linear congruence
Pencils of screws and linear congruencesSlide74
x
±
2
Hyperbolic linear congruences
»Slide75
x
»
1
p
1
±
1
±
2
p
2
Hyperbolic linear congruences
»
= (x
T
[ p
1
p
2
T
]x)
»
1
+ (x
T
[ p
1
p
2
T
]x)
»
2
+ (x
T
[ p
1
p
2
T
]x)
»
3
+ (x
T
[ p
1
p
2
T
]x)
»
4
»
2
»
3
»
4
»Slide76
x
»
1
p
1
±
1
±
2
p
2
Hyperbolic linear congruences
»
= (y
T
[ p
1
p
2
T
] y)
»
1
+ (y
T
[ p
1
q
2
T
] y)
»
2
+ (y
T
[ q
1
p
2
T
] y)
»
3
+ (y
T
[ q
1
q
2
T
] y)
»
4
y = u
1
y
1
+ u
2
y
2
+ u
3
y
3
= Y u
»
2
»
3
»
4
»
ySlide77
x
»
1
p
1
±
1
±
2
p
2
Hyperbolic linear congruences
»
=
(
u
T
[
¼
1
¼
2
T
]
u
)
»
1
+
(
u
T
[
¼
1
ρ
2
T
]
u
)
»
2
+
(
u
T
[
ρ
1
¼
2
T
]
u
)
»
3
+
(
u
T
[
ρ
1
ρ
2
T
]
u
)
»
4
= X û ,
where
X
2
R
6
£
4
and
û
2
R
4
»
2
»
3
»
4
»
ySlide78
x
ξ
±
a
2
p
1
z
p
2
p
a
1
Parabolic linear congruences
±
s
°
T
»
= X û ,
where
X
2
R
6
£
5
and
û
2
R
5Slide79
Elliptic linear congruences
x
»
y
»
= X û ,
where
X
2
R
6
£
4
and
û
2
R
4Slide80
x
»
1
y
1
»
2
y
2
Epipolar geometry
(
»
1
|
»
2
) = 0
or
û
1
T
F û
2
= 0,
where F = X1TX2 2
R4£4
Feldman et al. (2003): 6£
6 F for crossed-slit camerasGupta & Hartley (1997): 4£
4 F for linear pushbroom camerasSlide81
Trinocular geometry
Di (»1 , »2 , »3 ) = 0 or Ti
(û
1
, û
2 , û3 ) = 0, for i = 1,2,3,4
1 1 12
2 23
3
3
4
4 45 5 56 6
6
δ
η
φ
xSlide82
Admissible maps and
intrinsic parameters(Batog, Goaoc, Ponce, CVPR’10)
Optics
Retina
x
y’
l
l’
y
l
Because light travels along straight lines in homogeneous media, the
lines associated with any camera must form a congruence of order 1
(Sturm, 1893;
Beni
ć
and
Gorjanc
, 2006).Slide83
Proposition:
Given two cameras with the same underlying congruence but distinct retinas, the image formed by the first camera is projectively equivalent to the image formed by the other one after some projective transformation of space.In plain English:
The retinal plane matters.Slide84
Retina
x
l
y
Ax
A x
2Slide85
Retina
x
l
y
Ax
A x
2
Proposition:
A necessary and sufficient condition for a 4x4 matrix
A to be admissible—that is, induce a linear camera, is that its
minimum polynomial has degree 2.
Proposition:
There is a
bijection
between admissible maps and linear
cameras.Slide86
Intrinsic parameters
Hyperbolic
Parabolic
EllipticSlide87
Building a parabolic camera
(Batog, Goaoc, Lavandier, Ponce, 2010)