3D Geometric Reasoning Jiyan Pan 1232012 Task Detect objects Identify surface regions Estimate ground plane Infer gravity direction Geometrically coherent in the 3D world 3D geometric context ID: 626635
Download Presentation The PPT/PDF document "Coherent Scene Understanding with" is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
Coherent Scene Understanding with 3D Geometric Reasoning
Jiyan
Pan
12/3/2012Slide2Slide3
Task
Detect objects
Identify surface regions
Estimate ground plane
Infer gravity direction
Geometrically coherent in the
3D world
3D geometric contextSlide4
x
b
d
b
d
t
γ
n
v
θ
x
t
n
p
h
p
n
g
α
H
f
ground plane
image plane
(inverse) gravity
ground plane orientation
ground plane height
object vertical orientation
real world height
object depth
camera center
focal length
object pitch and roll angles
object
landmarks
Coordinate system
Deterministic relationships
Variables of global 3D geometries:
n
g
,
n
p
, h
pSlide5
x
b
d
b
d
t
γ
n
v
θ
x
t
n
p
h
p
n
g
α
H
f
ground plane
image plane
(inverse) gravity
ground plane orientation
ground plane height
object vertical orientation
real world height
object depth
camera center
focal length
object pitch and roll angles
object
landmarks
Coordinate system
Probabilistic relationships
Derived from appearance
Prior knowledgeSlide6
Can we solve them all for a coherent solution?
Non-linear
Non-deterministic
Even invalid equations from false detectionsSlide7
√
√
√
√
X
Global 3D context
Local 3D
contextSlide8
√
√
√
√
X
“Chicken and egg” problem:
Local entities could be
validated by global 3D context
Global 3D context is induced
from local entities
Global 3D context
Local 3D
context
?Slide9
Possible solution (All in PGM)
Put both global 3D geometries and local entities in a PGM
[1]
Precision issue: Have to quantize continuous variables
Complexity issue: Pairwise potential would contain up to ~1e6 entries
[1] D. Hoiem, A. A.
Efros, and M. Hebert. Putting objects in perspective. IJCV, 2008
Ground
o1
o
2
o
k
Gravity
100(pitch) × 100 (roll) × 100 (height)Slide10
Possible solution (Fixed global geometries as hypotheses)
Task much easier under a fixed hypothesis of global 3D geometries
Ground
o
1
o
2
o
k
Gravity
×
×
×
×
×
×Slide11
Task much easier under a fixed hypothesis of global 3D geometries
Possible solution
(Fixed global geometries as hypotheses)
o
1
o
2
o
k
ω
1
ω
2
ω
3
How to generate global 3D geometry hypotheses?Slide12
Possible solution(Hypotheses by exhaustive search)
Exhaustive search over the quantized space of global 3D geometries
[2]
Computational complexity tends to limit search space
[2] S.
Bao
et al. Toward coherent object detection and scene layout understanding. IVC, 2011Slide13
Possible solution
(Hypotheses by Hough voting)
Each local entity casts vote to the Hough voting space of the global 3D geometries and peaks are selected
[3]
False detections could corrupt the votes
Would applying EM help? Not likely, if false detections overwhelm
[3] M. Sun et al.
Object detection with geometrical context feedback loop. BMVC, 2010
L1
L
2
L
3
L
5
L
4
L
7
L
6Slide14
Our solution
We take a RANSAC-like approach: Randomly mix the contributions of local entities
L
1
L
2
L
3
L
5
L
4
L
7
L
6Slide15
Our solution
We take a RANSAC-like approach: Randomly mix the contributions of local entities
L
1
L
2
L
3
L
5
L
4
L
7
L
6Slide16
Our solution
We take a RANSAC-like approach: Randomly mix the contributions of local entities
Compared to averaging over all local entities:
More robust against outliersCompared to directly using estimates from each single local entity: More robust against noise
L
1
L2
L
3
L
5
L
4
L
7
L
6Slide17Slide18Slide19Slide20Slide21Slide22
0
5
10
15
20
25
30
35
40
45
50
1.6
1.8
2
2.2
2.4
2.6
2.8
3
Number of random mixtures
Minimum hypothesis error
Gravity Direction
Individual
Mixture
AverageSlide23
0
5
10
15
20
25
30
35
40
45
50
1.6
1.8
2
2.2
2.4
2.6
2.8
3
3.2
Number of random mixtures
Minimum hypothesis error
Ground Plane Orientation
Individual
Mixture
AverageSlide24
√
√
√
√
X
Local 3D
context
Global 3D contextSlide25
3D geometric context
ground plane orientation
valid
valid
invalid (#1)
invalid (#1)
invalid (#1)
ground plane
#1: Common ground (global)Slide26
3D geometric context
#2: Gravity direction (global)
(inverse) gravity
ground plane orientation
invalid (#2)
ground planeSlide27
3D geometric context
#3: Depth ordering (local)
(inverse) gravity
ground plane orientation
incompatible (#3)
ground planeSlide28
3D geometric context
#4: Space occupancy (local)
(inverse) gravity
ground plane orientation
incompatible (#4)
ground planeSlide29
2
3
4
5
6
1Slide30
2
3
4
5
6
1
Global geometric compatibility for an object:
Orientation:
Given a global 3D geometry hypothesis Slide31
2
3
4
5
6
1
Global geometric compatibility for an object:
Orientation:
Height:
Given a global 3D geometry hypothesis Slide32
2
3
4
5
6
1
Global geometric compatibility for a surface:
Orientation: local estimates vs. or
Location: horizontal surface region vs. ground horizon
Given a global 3D geometry hypothesis Slide33
2
3
4
5
6
1
Local geometric compatibility for two objects:
Depth ordering:
Space occupancy:
Given a global 3D geometry hypothesis Slide34
2
3
4
5
6
1
Objective function of the CRF:
Given a global 3D geometry hypothesis Slide35
√
√
√
√
X
Local 3D
context
Global 3D context
Best hypothesisSlide36
3D reasoning agrees with raw detector
3D reasoning recovers detection rejected by raw detector
3D reasoning rejects detection accepted by raw detector Slide37
3D reasoning agrees with raw detector
3D reasoning recovers detection rejected by raw detector
3D reasoning rejects detection accepted by raw detector Slide38
3D reasoning agrees with raw detector
3D reasoning recovers detection rejected by raw detector
3D reasoning rejects detection accepted by raw detector Slide39
3D reasoning agrees with raw detector
3D reasoning recovers detection rejected by raw detector
3D reasoning rejects detection accepted by raw detector Slide40
3D reasoning agrees with raw detector
3D reasoning recovers detection rejected by raw detector
3D reasoning rejects detection accepted by raw detector Slide41
3D reasoning agrees with raw detector
3D reasoning recovers detection rejected by raw detector
3D reasoning rejects detection accepted by raw detector Slide42
3D reasoning agrees with raw detector
3D reasoning recovers detection rejected by raw detector
3D reasoning rejects detection accepted by raw detector Slide43
3D reasoning agrees with raw detector
3D reasoning recovers detection rejected by raw detector
3D reasoning rejects detection accepted by raw detector Slide44
0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
5
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
False Positive per Image
True Positive Rate
Deformable Part Model Detector
Baseline
Hoiem
Ours
3D geometric reasoning improves object detection performance
D
.
Hoiem
, A. A.
Efros
, and M. Hebert. Putting objects in perspective. IJCV, 2008Slide45
0
0.2
0.4
0.6
0.8
1
1.2
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
False Positive per Image
True Positive Rate
Dalal-Triggs
Detector
Baseline
Hoiem
Ours
3D geometric reasoning improves object detection performance
D
.
Hoiem
, A. A.
Efros
, and M. Hebert. Putting objects in perspective. IJCV, 2008Slide46
Improvement in AP over baseline detector
Ours
10.4%
Hoiem
4.8%
Sun
5.1%
M. Sun
et al
. Object detection with geometrical context feedback loop. BMVC, 2010
D. Hoiem, A. A. Efros
, and M. Hebert. Putting objects in perspective. IJCV, 20083D geometric reasoning improves object detection performanceSlide47
Horizon estimation
median error
Ours
2.05
⁰
Hoiem
3.15
⁰
Sun
2.41⁰
M. Sun et al. Object detection with geometrical context feedback loop. BMVC, 2010
D. Hoiem, A. A. Efros, and M. Hebert. Putting objects in perspective. IJCV, 2008Slide48
√
√
√
√
X
Local 3D
context
Global 3D context
Best hypothesisSlide49
Contributions of different geometric context
0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
5
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
False Positive per Image
True Positive Rate
Detection ROC Curve
Det
Det+IdvlGeo
Det+PairGeo
Det+FullGeoSlide50
Benefit is mutual
Error in
gravity direction
Error in
ground
orientation
Vanishing points alone
2.62⁰
4.85
⁰
Whole system2.05
⁰
2.21⁰Slide51
Extensions
Improved depth ordering constraint
Local geometric constraints involving vertical surfaces
Multiple supporting planes
Using more prior knowledge of objects
Utilizing semantic categories of surface regionsSlide52Slide53
closer object
farther object
closer object
farther object
occlusion mask of the farther object
intersection region of the two object masks
√
X
Fully cover?
Fully cover?Slide54
Occlusion: bottleneck in our system
Missed detection
Erroneous estimation of local properties
Less effective depth ordering constraintSlide55
Generalized Hough voting: better at handle occlusions
K.
Rematas
et al
. CORP 2011
B.
Leibe
et al. IJCV 2008Slide56Slide57
Occlusion-and-geometry-aware Hough votingSlide58
√
√
√
√
X
Local 3D
context
Global 3D context
Best hypothesisSlide59
So far we have treated the entire region labeled as "vertical" as a wholeSlide60
Decompose vertical region into surface segments
Occlusion boundary recovery (
Hoiem
et al. IJCV’11)
Vanishing line sweeping (Lee et al. CVPR’09)Slide61Slide62
ground plane
inverse gravity
√
vertical surface candidate 1
vertical surface candidate 2Slide63
ground plane
vertical surface candidate 1
inverse gravity
vertical surface candidate 2
XSlide64Slide65
ground plane
vertical surface candidate
inverse gravity
object candidate
√Slide66
object candidate
ground plane
vertical surface candidate
inverse gravity
XSlide67
Given object layout, erect surfaces one by one
“Interpretation by synthesis” (Gupta et al. ECCV’10)Slide68Slide69
supporting plane 1Slide70
supporting plane 1
supporting plane 2Slide71
ground planeSlide72
w
l
βSlide73Slide74
Spring 2013 (ICCV’13 submission)Improved depth ordering constraint
Using more prior knowledge of objects
Multiple supporting planes
Fall 2013 (CVPR’14 submission)
Local geometric constraints involving vertical surfaces
Utilizing semantic categories of surface regionsDuring Spring Semester of 2014Thesis writingSlide75
Expected Contributions
Systematically model the relationships among global and local geometric variables
Develop a RANSAC-CRF scheme to handle non-linear, non-deterministic, and possibly invalid relationships
Occlusion-and-geometry-aware object detection for finer depth order reasoning
Joint reasoning among global geometries, surface segments, and objectsSlide76
Thank you!