Segmentation from Examples - PowerPoint Presentation

Slide1

Segmentation from Examples

By: A’laa

Kryeem

Lecturer:

Hagit

Hel-OrSlide2

What is Segmentation from Examples

?

Segment an image based on one (or more) correctly segmented image(s) assumed to be from the same

domainEffective when making a semantic segmentationSlide3

Why to use Examples

The example defines the granularity of the desired

output

Give us the ability to characterize meaningful parts in the imageUsing example allow us to use non-parametric modelSlide4

The example defines the granularity of the desired

output:

Training image

Test Image

Desired Segmentation

Induced SegmentationSlide5

Why to use Examples

The example defines the granularity of the desired

output

Give us the ability to characterize meaningful parts in the imageSlide6

Give us the ability to characterize meaningful parts in the imageSlide7

Semantic Segmentation from an example

We want to segment an image into semantically meaningful

parts

Required in various applicationsSlide8

Semantic Segmentation from an example

We want to segment an image into semantically meaningful

parts

Required in various applicationsProblems: Meaningful parts are often too complexSemantic interpretation is highly subjective, depending on both the application, and the userSlide9

Meaningful parts are often too complexSlide10

Semantic Segmentation from an example

We want to segment an image into semantically meaningful

parts

Required in various applicationsProblems: Meaningful parts are often too complexSemantic interpretation is highly subjective, depending on both the application, and the userSlide11

Example of image different segmentationSlide12

Semantic Segmentation from an example

So, How to achieve semantic segmentation

Getting segmented training image(s) as

inputSlide13

Training setSlide14

Semantic Segmentation from an example

So, How to achieve semantic segmentation

Getting segmented training image(s) as

inputUsing non-parametric representationSlide15

non-parametric model

Each semantic part is represented by a set of square patchesSlide16

Semantic Segmentation from an example

So, How to achieve semantic segmentation

Getting segmented training image(s) as

inputUsing non-parametric representationOver-segmenting the Test image into small fragmentsSlide17

Over segmented imageSlide18

Semantic Segmentation from an example

So, How to achieve semantic segmentation

Getting segmented training image(s) as

inputUsing non-parametric representationOver-segmenting the Test image into small fragmentsCompute costs for fragment-label pairsSlide19

(fragment,label

) cost example

?Slide20

Semantic Segmentation from an example

So, How to achieve semantic segmentation

Getting segmented training image(s) as

inputUsing non-parametric representationOver-segmenting the Test image into small fragmentsCompute costs for fragment-label pairs

Graph-cuts multi-label optimizationSlide21

Why do we need graph-cuts

Graph-cuts optimization

is used

to label each fragment in a globally optimal mannerSlide22

Training set

Test image

Fragmentation

Fragments

Patch sets

Classification

Classification scores

Graph-Cuts optimization

ResultSlide23

Over segmenting

Fragment: small arbitrarily-shaped and

simply-connected pixel

clustersWe assume that small homogeneous regions always belong to the same semantic partSlide24

Over segmenting

Fragment: small arbitrarily-shaped and simply connected pixel

clusters

We assume that small homogeneous regions always belong to the same semantic partAdvantages:Enforces a locally coherent labelingReduces the computational complexitySlide25

Graph-cuts multi-label optimization

For each fragment we have k cost values, we need to determine which one is the

optimal

Using expanded version of the graph-cuts we saw at Jad’s lecture, where we may have more than two labels (background , object)Slide26

Algorithm for semantic segmentation

Pixel labeling

costs

FragmentationFragment labeling costsGraph-cuts optimizationSlide27

Algorithm for semantic segmentation

Pixel labeling

costs

FragmentationFragment labeling costsGraph-cuts optimizationSlide28

Pixel labeling costs

Given

I

train and Ltrain Representing each label(segment) in Ltrain by a set of square patches

We get k sets {Sl} l=1,…,k one for each

labelSlide29

Pixel labeling costs (cont.)

Next, we define

φ (

p, l) for each (pixel,label) pairThe cost of assigning label l to pixel p I

test

= min

P’



S

l

p:pixel

at

I

test

l : label in

L

train

ssd

(P,P’) is the sum of squared distances between P,P

’

M:mxmx3

P:mXm neighborhood centered at

p

P’:

mxm

patchSlide30
Slide31

Algorithm for semantic segmentation

Pixel labeling

costs

FragmentationFragment labeling costsGraph-cuts optimizationSlide32

Fragmentation

We partition

I

test

into

small,color

-homogeneous

regions using mean shift segmentationSlide33

Fragmentation

We partition

I

test

into

small,color

-homogeneous

regions

using mean shift

segmentation

Fragment size is adjusted according to

I

test

.(fragments

are smaller in more detailed areas of

I

test

, and larger in more homogeneous regions

)Slide34

Fragmentation

We partition

I

test

into

small,color

-homogeneous

regions

using mean shift

segmentation

Fragment size is adjusted according to

I

test

.(fragments are smaller in more detailed areas of

I

test

, and larger in more homogeneous regions)

Fragment boundaries align with edges in the

imageSlide35

Fragmentation (cont.)

Random colorization

Detailed close-upSlide36

Algorithm for semantic segmentation

Pixel labeling

costs

FragmentationFragment labeling costsGraph-cuts

optimizationSlide37

Fragment labeling costs

Voting scheme in order to compute labeling costs of each

fragment

For each fragment fItest we pick a few representative pixels: Rep(f)={pi f }

i=I,…,Rf R

f

is proportional to |f

|Slide38

Fragment labeling costs (cont.)

We talked about the cost of assigning a pixel into a label

Rep(f)}

 Slide39

Fragment labeling vs. pixel labeling

Fragment labeling reducing complexity

We have n=

Assume |Rep(f)|=

,then we need to compute costs only for

pixels

 Slide40

Fragment labeling vs. pixel labeling

Enforces a locally coherent

labeling

Training image

Training

seg

.

Input image

Fragment labeling

Pixel labelingSlide41

Algorithm for semantic segmentation

Pixel labeling

costs

FragmentationFragment labeling costs

Graph-cuts optimizationSlide42

Graph-cuts optimization

After calculating labeling cost for all image fragments we get k images. Image

i

describes the cost assigning each pixel at the test image to label ifragment labeling costs. Costs range in the interval [0,1]Slide43

Graph-cuts optimization

Now for each pixel

p

Itest we have a labeling costWe need to find Ltest the globally optimal labeling

Requirements:Minimizes the total labeling cost

Consistent

with

presence (or absence) of

edgesSlide44

Graph-cuts optimization (cont.)

For each pair of neighboring pixels

we define:

Ψ

(p,q,L(p),L(q))=

L(p),L(q) : labels assigned to

p,q

: difference between pixels

p,q

(RGB

euclidian

distance

)

 Slide45

Graph-cuts optimization (cont.)

In order to force pixels within each fragment to be labeled the same, and reduce complexity we specify the energy term

E(L) (E(L) the value of a labeling

scheme) in terms of fragments instead of pixels :

: cost of

assigning

fragment f to label L(f

), weighted

by the size of each fragment.

: neighboring fragments in

I

test

.

=

:controls trade-off between regions and boundaries

 Slide46

Intuition:

Big

value:

For pixels p,q even with different colors the graph-cuts step prefers to connect them to the same label to have

and reduce the energy. Instead of 1-

althout

can be big too.

This mean we prefer continues regions and not edges.

 Slide47

Intuition:

s

value:

For pixels

p,q even with similar colors the graph-cuts step won’t care about connecting

them to

different labels because of small

value. Even with big value at1-

multiplying

still give us small E value.

Favors boundaries,

holding out non-continues regions .

 Slide48

Graph-cuts optimization (cont.)

Finally

Ltest is determined by solving

Ltest=minL E(L)

Fragment labeling

Labeling after Graph-Cuts OptimizationSlide49

Multi-label graph-cut

Colored

nodes:labels

Squares : fragments

For each (

fragment,label

) pair we have an edge.

Edge weigh according to

φ

.

Edges between two squares weighed according to

Ψ

. Slide50

Multi-label graph-cut

Induced graph

Each fragment connected to a single label. Slide51

Multi-label graph-cut is NP-complete problem!

Using Isolation Heuristic we

can get an

approximation of E(L) For 1i

k construct a minimum weight isolating cut

E

i

for label L

i

.

Determine h such that E

h

has max weight.

E=

.

Return E.

 Slide52

effect

Training image

Training segmentation

Input image

=0.1

=5

=1

 Slide53

Algorithm for semantic segmentation

Pixel labeling

costs

FragmentationFragment labeling costs

Graph-cuts optimizationSlide54

Algorithm results

Training set

a

b

cSlide55

Bear results

invariant to the number of instances of each semantic part within the image, and insensitive to the shape of each part.

We can’t separate multiple objects belonging to the same label (c).Slide56

Algorithm results

Training set

a

b

c

dSlide57

Summary

We saw that giving segmented example from the same domain of an image can effectively perform a semantic

segmentation

Using example also defines the granularity of the desired outputdetermining whether an entity belongs to a particular semantic part is more easily done at the fragment level, than on a pixel-by-pixel basisUsing graph-cuts with multi-label support can help making global optimization step for finding optimal labeling

Only one parameter needed , controlling the trade-off between regions and

boundaries

 Slide58

Thank You For ListeningSlide59

References

Inducing Semantic Segmentation from an Example,

Yaar

 Schnitman, Yaron Caspi, Daniel Cohen-Or, and Dani Lischinski."Segmentation by Example“, Sameer Agarwal and Serge Belongie.

Christoudias, C.M., Georgescu, B.: Edge detection and image segmentation (edison) system.

Boykov

, Y.,

Veksler

, O.,

Zabih

, R.: Fast approximate energy minimization

via graph

cuts. IEEE Trans. Pattern Anal. Mach.

Intell. 23

(2001) 1222–1239