Inferring Temporal Order of Images from 3D Structure - PowerPoint Presentation

Slide1

Inferring Temporal Order of Images from 3D Structure

Grant Schindler Gatech

Frank Dellaert Gatech

Sing Bing Kang MSR, Redmond

Slide2

Outline

Problem Definition

Algorithm Overview

Applications

Things to think about

Slide3

What can be done with n images?

Slide4

What can be done with n images?

Feature Extraction

Correspondence

Structure from Motion

What Now?

Slide5

Temporal Ordering and 4D Walkthrough

1920

1951

1966

2003

Slide6

Outline

Problem Definition

Algorithm Overview

Applications

Things to think about

Slide7

SFM tells us:

Camera Matrices

3D points for features

Visibility of 3D points in images

C2

F3

C1

I1

I2

F1

F2

Slide8

SFM info

Image 1 (I1)

Image 2 (I2)

F1

Visible

???

F2

???

Visible

F3

Visible

???

C2

F3

C1

I1

I2

F1

F2

Slide9

SFM info

Image 1 (I1)

Image 2 (I2)

F1

Visible

???

F2

Occluded

Visible

F3

Visible

Out

of View

C2

F3

C1

I1

I2

F1

F2

Slide10

SFM info

Image 1 (I1)

Image 2 (I2)

F1

Visible

???

F2

Occluded

Visible

F3

Visible

Out

of View

Notion of missing at that time

C2

F3

C1

I1

I2

F1

F2

Slide11

Classification of 3D point for an Image

Visible

– SFM tells us

Out of View

– Camera Matrix tells us

Missing / Occluded

- ???

for an occluded point, there must be an occluder

Slide12

Point ‘F’ Missing / Occluded ?

Find out occluders

3D Triangulation of all visible points

No occluder should occlude a visible point

Visibility check for F

occluders

F1

F2

Camera centre

occluded

missing

Slide13

Visibility Matrix

I1

I2

...

In

F

1

S

11

S

12

…S1n

F2S21

S22…

S2n…

…

…

…

…

F

m

S

m1

S

m2

…

S

mn

S

ij

€

{visible, occluded, missing, out of view }

Slide14

Constraints of Visibility Matrix

Slide15

Combinatorial Algorithm to find Best Configuration

Local search method

Starts at a random configuration

Small moves which reduce the number of constraints violated

Slide16

Issues leading to Finding

Approximate Solution

Problems in feature detection

Mislabeling of points

Triangulation strategy

Inaccuracy in SFM

Features occluded by undetected occluders (fog, trees etc)

Slide17

Structural Segmentation from Temporal Ordering

Clustering temporally coherent features

Separate triangulation of each cluster

Texture by projecting on images

Slide18

Algorithm Overview

Slide19

Possible Applications

Historic Preservation

Virtual Tourism

Urban Planning

Spatio-Temporal models as a new way

to

interact

with a vast collection of imagery

Slide20

Things to Think about

Feature extraction (done manually here)

Better methods for finding occluders – problems with triangulation method

Very coarse structure

Can have triangles for no occluders

Using Goesele’s work (ICCV 2007) for structural segmentation

High number of images required (this paper used 20-30 images)

Validation

Correspondence between the best ground truth solution and best approximate solution of ordering

Increasing the scale technically and physically

Slide21

An Interesting Insight….

Assume no building can be demolished once it’s built

Assume every image is a node of graph

Edge from A to B if A precedes B

(B has visible features missing in A )

Directed Graph (acyclic in ideal case)

Slide22

C

A

B

Directed Graph (Acyclic in ideal case)

B1

B2

B3

B2

B2

B3

A

B

C

Input Images

Slide23

C

A

B

Directed Graph (Acyclic in ideal case)

B1

B2

B3

B2

B2

B3

A

B

C

Input Images

B

C

A

Topological Sort

Solution !

Slide24

More insights about Graph Model

Every edge has a confidence value based on quality of features and SFM procedure

In general, there can be back edges in this graph

Problem to find the best topological sort maximizing the confidence measure

Slide25

Graph Complexity

Increases with more constraints

Modeling constraints involving more than 2 images at a time -

how??