Shape From Shading - PowerPoint Presentation

Slide1

Shape From Shading

Mica Arie-Nachimson

May 2011Slide2

What is Shading?

Well… not shadow…

We can’t reconstruct shape from one shadow…

Image from www.moolf.comSlide3

What is Shading?

Variable levels of darkness

Gives a

cue for

the actual 3D shape

There is a relation between intensity and shape

Images from S.

Narasimhan

, Carnegie Mellon; www.erco.com; www.alesmav.com; H. Wang, University of CaliforniaSlide4

Shading Examples

These circles differ only in grayscale intensity

Intensities

give a strong “feeling” of scene structureSlide5

Talk Outline

Introduction and BasicsNotationsReflectance map

Photometric stereo

Main Approaches

Propagation Solutions

Basic Solution by Horn [85]Using Fast Marching, Kimmel & Sethian [2001]

Global energy minimization solution using Belief Propagation – only if time permitsSlide6

Problem definitionNotations

Reflectance MapPhotometric Stereo

Introduction and BasicsSlide7

What determines scene radiance?

The amount of light that falls on the surface

The fraction of light that is reflected (albedo)

Geometry of light reflection

Shape of surface

Viewpoint

n

 Slide8

Albedo

A property of the surface

Usually normalized

Can be modeled as a single scalar for the entire surface or dependent on location

Usually assumed to be a known scalar

Is this a ball with uniform albedo or is this a 2D circle with varying albedo?

We’re not going to talk about the albedo

 Slide9

Surface Normal

Convenient notation for surface orientation

A smooth surface has a tangent plane at every point

We can model the surface using the normal at every pointSlide10

The Shape From Shading Problem

Given a grayscale imageAnd albedo

And light source direction

Reconstruct scene geometry

Can be modeled by surface

normalsSlide11

Lambertian Surface

Appears equally bright from all viewing directionsReflects all light without absorbing

Matte surface, no “shiny” spots

Brightness

of the surface as seen from

camera is linearly correlated to the amount of light falling on the surface

n

Today we will discuss only

Lambertian

surfaces under point-source illuminationSlide12

Some Notations:Surface Orientation

A smooth surface has a tangent plane at every point

Mark

Parametrize

surface orientation by first partial derivatives of

 Slide13

Some Notations:Surface Orientation

Surface normal

,

Normalize

 Slide14

Bi-Directional Reflectance Distribution

BRDFHow bright a surface appears when

Viewed from one direction

Light falls on the surface from

another direction

n

Amount of light falling on the surface

The brightness of the surface as seen from cameraSlide15

Lambertian Surface

Appears equally bright from all viewing directionsReflects all light without absorbing

Brightness

of the surface as seen from

camera is linearly correlated to the amount of light falling on the surface

n

 Slide16

Reflectance Map

Relationship between surface orientation and brightness

Lambertian

surface

Image irradiance (brightness) is proportional to

 Slide17

Reflectance Map Example

Brightness as a function of surface orientation

iso-brightness

contour

Illustration

from

S.

Narasimhan

, Carnegie

Mellon

Lambertian

surface

 Slide18

Reflectance Map of a Glossy Surface

Brightness as a function of surface orientation

Surface with diffuse and glossy componentsSlide19

Reflectance Map Examples

Brightness as a function of surface orientationSlide20

Graphics with a 3D Feel

Given a 3D surface

, lighting and viewing direction, we can compute the gray level of pixel

of the surface

Find the gradient of the surface

Use

to determine the gray level

 Slide21

Shape From Shading?

Can we reconstruct shape from a single image?

Two variables

with one equation… So what do we do?

Illustration

from

S.

Narasimhan

, Carnegie

MellonSlide22

Shape From Shading!

Use more imagesPhotometric stereo

Shape from shading

Introduce constraints

Solve locally

Linearize problem

Images from

R.

Basri

et

al. “Photometric Stereo with General, Unknown

Lighting“,

IJCV 2006Slide23

Photometric Stereo

Take several pictures of same object under same viewpoint with different lightingSlide24

Photometric Stereo

Take several pictures of same object under same viewpoint with different lightingSlide25

Photometric Stereo

Take several pictures of same object under same viewpoint with different lightingSlide26

A Quick Review

Main Approaches forShape From ShadingSlide27

Main Approaches

Minimization: Solve for a global error function while introducing constraint(s):

Brightness

Smoothness

Intensity gradient

…

 Slide28

Main Approaches

Minimization: Solve for a global error function while introducing constraint(s):

Brightness

Smoothness

Intensity gradient

…

Propagation: Grow a solution around an initial known point or boundary

Local: Assume local surface type

Linear: Make problem linear by a linearization of reflectance function

 Slide29

Main Approaches

Minimization: Solve for a global error function while introducing constraint(s):

Brightness

Smoothness

Intensity gradient

…

Propagation: Grow a solution around an initial known point or boundary

Local: Assume local surface type

Linear: Make problem linear by a linearization of reflectance function

 Slide30

Main Approaches

Minimization: Solve for a global error function while introducing constraint(s):

Brightness

Smoothness

Intensity gradient

…

Propagation: Grow a solution around an initial known point or boundary

Local: Assume local surface type

Linear: Make problem linear by a linearization of reflectance function

 Slide31

Horn [85]Solution by Characteristic Curves

Basic Propagation SolutionSlide32

Propagating Solution

Suppose that we know the depth of some point on the surface

Extend the solution by taking a small step

But

and

are unknown, and the image irradiance equation gives only one constraint

If we knew

and

, can compute changes in

using second partial derivatives

and

 Slide33

Propagating Solution

, what else can we use?

Image irradiance equation

 Slide34

Propagating Solution

;

We note that for a small step

For these specific

If you take a small step in the image plane parallel to the direction of the gradient of

,

then

can

compute change in

 Slide35

Propagating Solution

;

Get ODEs:

;

;

;

;

These

equations describe a characteristic curve

 Slide36

Propagating Characteristic Curve

Characteristic curve

Need to initialize every curve at some known point

Singular points

Occluding boundaries

Image

by

Dejan

TodorovićSlide37

Propagating Characteristic Curve

Characteristic curve

Need to initialize every curve at some known point

Singular points

Occluding boundaries

Curves are “grown” independently, very instable

Image

by

Dejan

Todorović

 Slide38

“Optimal Algorithm for Shape from Shading and Path Planning”, R. Kimmel and J. A. Sethian

[2001]

Shape From Shading by Fast MarchingSlide39

Vertical Light Source Case

Recall reflectance map

Assume light source located near the viewer

=(0,0,1)

This is an

Eikonal

equation

Can be solved by an

numerical algorithm

Fast MarchingSlide40

Fast Marching

Expanding Dijkstra’s shortest path algorithm for general surfacesRepresented as triangulated mesh

Flat domains

Many computer vision problems can be set into flat domains

Every pixel is a node, edges

between adjacent nodesSlide41

Dijkstra’s Shortest Paths

Connected graphsStart with the source node

s

7

15

10

9

14

9

2

11

6Slide42

Dijkstra’s Shortest Paths

Connected graphsStart with the source node

Update all neighbors

s

7

9

14

7

15

10

9

14

9

2

11

6Slide43

Dijkstra’s Shortest Paths

Connected graphsStart with the source node

Update all neighbors

Go to the closest neighbor

Set its value

s

7

9

14

7

15

10

9

14

9

2

11

6Slide44

Dijkstra’s Shortest Paths

Connected graphsStart with the source node

Update all neighbors

Go to the closest neighbor

Set its value

Compute its neighborsUpdate smaller scores

s

7

9

14

23

7

15

10

9

14

9

2

11

6Slide45

Dijkstra’s Shortest Paths

Connected graphsStart with the source node

Update all neighbors

Go to the closest neighbor

Set its value

Compute its neighborsUpdate smaller scoresContinue with smallest value nodeRemember path

s

7

9

14

23

7

15

10

9

14

9

2

11

6Slide46

Flat Domains: Why Does Dijkstra Fail?

Dijkstra will not find the

diagonal path

Need to examine trianglesSlide47

Fast Marching: Problem Definition

Suppose there is a forest fire with multiple sourcesEvery point that was touched by the fire is burnt and will not be visited by the fire again

Firemen

register the time

T

(x) at which the fire arrives to location x.The fast marching algorithm simulates this scenarioSlide48

Fast Marching: Problem Definition

Multiple sources

Advancing forward

Advances either at a constant rate

or at varying rate

What is the arrival time at every location?

Image from G.

Rosman

,

TechnionSlide49

Fast Marching Algorithm

Set

and mark the points as

done

Set the rest of the points as

and mark them as

far

All

far

points adjacent to

done

points become

verify

pointsUpdate all verify points using

of the done setThe

verify

with the smallest

becomes

done

Continue until all points are

done

Illustrations from selected publications of J. A.

SethianSlide50

Fast Marching on a Grid

Update step

T

i,j-1

i,j+1

i-1,j

ij

i+1,j

i+1,j

i,j+1

T

T

T

T

i-1,j

i,j-1

ij

?

Slide based on slides by R. Kimmel,

TechnionSlide51

Fast Marching on a Grid

Update step

T

i,j-1

i,j+1

i-1,j

ij

i+1,j

i+1,j

i,j+1

T

T

T

T

i-1,j

i,j-1

ij

?

Slide based on slides by R. Kimmel,

TechnionSlide52

Fast Marching on a Grid

Update step

T

i,j-1

i,j+1

i-1,j

ij

i+1,j

i+1,j

i,j+1

T

T

T

T

i-1,j

i,j-1

ij

?

Slide based on slides by R. Kimmel,

TechnionSlide53

Fast Marching on a Grid

Initialization: all

or known initial value

Update step:

Fit a triangle with gradient

and two determined values at

neighboring grid points

T

i,j-1

i,j+1

i-1,j

ij

i+1,j

i+1,j

i,j+1

T

T

T

T

i-1,j

i,j-1

ij

Slide based on slides by R. Kimmel,

TechnionSlide54

Update Step Details

Update step is a quadratic equation:

Solution:

Disregard the “minus” solution

which yields

 Slide55

Update Step Details

Update step is a quadratic equation:

Solution:

If

, then

and

 Slide56

Update Step Details

Update step is a quadratic equation:

Solution:

If

, then

and

 Slide57

Update Step Details

Update step is a quadratic equation:

Solution:

If

, then

and

and

This means that the wave front propagation comes from outside the triangle

Only choice is to update

 Slide58

Update Step Details

Update step is a quadratic equation:

If

Else

 Slide59

Shape From shadingVertical Light Source Case

Eikonal

equation

Numerical approximation of this equation

Assume

w.l.o.g

. that

 Slide60

Shape From ShadingVertical Light Source Case

Eikonal

equation

Numerical approximation of this equation

Assume

w.l.o.g

. that

 Slide61

Vertical Light Source CaseDerive Update Step

 Slide62

Vertical Light Source CaseDerive Update Step

Where:

If

Otherwise

 Slide63

Vertical Light Source CaseFast Marching

Fast marching introduces order to these update steps

Points are updated from small to large

Total complexity:

for the selection of smallest point and update of neighboring points

points (

pixles

)

 Slide64

General Light Source

Recall reflectance map

In this case

The right hand side depends on

This is not an

Eikonal

equation anymore

How can we solve this?

Solve in the light source coordinates, then change the variables backSlide65

General Light SourceChange Coordinate System

Before moving to light source coordinate system, choose

w.l.o.g

light source direction to be

, where

This will simplify our equations a bit

 Slide66

General Light SourceChange Coordinate System

Change coordinate system

Light source

coordinates marked

with

~

Eikonal

equation!

 Slide67

General Light SourceSolution in New Coordinate System

But…

We need to find

in order to find the intensity

Use the smallest

value from all neighbors of

 Slide68

General Light Source SolutionUpdate Step

Where:

If

Otherwise

 Slide69

Results

Results shown on synthetic shading image

 Slide70

“Efficient Belief Propagation for Vision Using Linear Constraint Nodes”, B. Potetz

[2007]

Minimization Using Efficient

Belief PropagationSlide71

Belief Propagation

Estimate the marginals

of a multivariate probability distribution

,

potential function

consists of several

elements

Usually represented as a

factor graph

 Slide72

Factor Graph

,

Bipartite graph

Every element of

is represented by a variable node

Every potential function

represented by a factor node

, connected to all variable nodes of

circles: variable nodes

s

quares: factor nodesSlide73

Belief Propagation on a Factor Graph

,

Estimate the

marginals

Iteratively compute messages along edges of the factor graph

Every variable node computes its marginal probability according to the probabilities its neighbors sent

Usually converges to a good enough local minimum

 Slide74

Shape From Shading Using aFactor Graph

Recall

Constraint I: Consistent integral

Where

Integral of the surface gradient along a closed curve should be zero

Illustration from Klaus & Horn, “Robot Vision”, MIT Press, 1986Slide75

Shape From Shading Using aFactor Graph

Constraint II:

Lambertian

constraint

Where

could be changed for other lighting conditions

Constraint III: Shape prior

M

odeled by a Laplace distribution

,

 Slide76

Shape From Shading Using aFactor Graph

Constraint I: Consistent integral

Constraint II:

Lambertian

constraint

Constraint III: Shape prior

,

o

ne pixel

o

ne pixel

o

ne pixel

o

ne pixel

o

ne pixel

Variable nodes

Factor node, consistent integral

Factor node,

Lambertian

constraint

Factor node, shape priorSlide77

Shape From Shading Using aFactor Graph

Constraint I: Consistent integral

Constraint II:

Lambertian

constraint

Constraint III: Shape prior

,

Variable nodes

Factor node, consistent integral

Factor node,

Lambertian

constraint

Factor node, shape priorSlide78

Results

Input:Result:

original surface

Rendering of original surface with

Input image

r

ecovered surface

Rendering of recovered surface with

 Slide79

Conclusion

Shape from shading problemDefinitionMain difficulties

Main approaches

Propagating solutions

Horn [85]

Kimmel [2001]Minimization with BP while introducing constraintsSlide80

Thank You!