Object Segmentation. Jaechul Kim and Kristen . Grauman. University of Texas at Austin. Problem statement. Category-independent object segmentation: . Generate object segments in the image . regardless of their categories.. ID: 142582
DownloadNote - The PPT/PDF document "Shape Sharing for" is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Presentations text content in Shape Sharing for
Slide1
Shape Sharing for Object Segmentation
Jaechul Kim and Kristen GraumanUniversity of Texas at Austin
Slide2
Problem statement
Category-independent object segmentation: Generate object segments in the image regardless of their categories.
Slide3
Related work
Top-down, category-specific approache.g., Active Contours (IJCV 1987), Borenstein and Ullman (ECCV 2002), Levin and Weiss (ECCV 2006), Kumar et.al. (CVPR 2005)
Use generic knowledge on object shapese.g., Levinshtein et.al. (ICCV 2009)
Category-independent multiple segmentationse.g., Malisiewicz and Efros (BMVC 2007), Carreira and Sminchisescu (CVPR 2010), Endres and Hoiem (ECCV 2010)
+ robustness to low-level cues- typically viewpoint specific- requires class knowledge!
How to model
top-down shape in a category-independentway?
horse shape priors
color, textures, edges…
Malisiewicz
and
Efros
(BMVC 2007),
Arbelaez et al. (CVPR 2009)Carreira and Sminchisescu (CVPR 2010)Endres and Hoiem (ECCV 2010)
Active Contours (IJCV 1987)Borenstein and Ullman (ECCV 2002) Levin and Weiss (ECCV 2006)Kumar et.al. (CVPR 2005)
e.g.,
e.g.,
Slide5
Our goal
Segment even unfamiliar objects with category-independent top-down cues
Top-down segmentation with shape prior
We don’t want to care what is in the image.
Cow? Sheep?
Slide6
Our idea:Shape sharing
Semantically close
Semantically disparate
Object shapes are shared among different categories.
Shapes from one class can be used to segment another (
possibly unknown
) class:
Enable category-independent shape priors
Slide7
Basis of approach:
transfer through matching
Transfer category-independent shape prior
Exemplar image
Test image
Partial shape
match
Global shape
projection
ground truth
object boundaries
Slide8
1. Shape projection
via local shape matches
…
Exemplars
Test image
Approach: Overview
+
Shape prior
Color model
Segmentation model per each group
2. Aggregating
the shape projections
Graph-cut
Segmentation hypotheses
3. Multiple figure-ground segmentations
with shape prior
Slide9
Approach: Shape projection
Test image
Exemplars
Projection
Aggregation
Segmentation
Vs.
BPLRs
Superpixels
Boundary-Preserving Local Regions (BPLR):
Distinctively shaped
Dense
Repeatable
[Kim & Grauman, CVPR 2011]
Slide10
Approach: Shape projection
Test image
Exemplars
…
…
Shape projections via
similarity transform of BPLR matches
Projection
Aggregation
Segmentation
Matched
Exemplar 1
Matched
Exemplar 2
Shape hypotheses
Slide11
Approach: Refinement of projections
Refined shape
Initial projection
Exemplar
jigsaw
Projection
Aggregation
Segmentation
Include
superpixels
where majority of pixels overlap projection
Align with bottom-up evidence
Slide12
Approach: Aggregating projections
Projection
Aggregation
Segmentation
…
Exploit
partial
agreement
from multiple exemplars
Grouping based on overlap
Slide13
+
Shape prior
Color model
Segmentation model per
each
group
Approach: Segmentation
Projection
Aggregation
Segmentation
Figure-ground segmentation using graph-cut
Slide14
n-links
Projection
Aggregation
Segmentation
Approach: Graph-cut
data term
smoothness term
Define a graph over image pixels:
node = pixel
edge = cost of a cut between pixels
Energy function to minimize:
Slide15
Projection
Aggregation
Segmentation
Approach:
Segmentation
Color likelihood
Fg
Bg
NA
Fg
color histogram
Bg
color histogram
Shape likelihood
Graph-cut optimization
Data term
+
Smoothness term
Slide16
Projection
Aggregation
Segmentation
Approach: Multiple segmentations
Parameter controlling
data term bias
Compute multiple segmentations
by varying
foreground bias:
…
Output:
Carreira
and
Sminchisescu
,
CPMC: Automatic Object Segmentation Using Constrained Parametric Min-Cuts
PAMI
2012
.
Slide17
Experiments
Exemplar database: PASCAL 2010 segmentation task training set (20 classes, 2075 objects)
Test datasets: PASCAL 2010 segmentation task validation set (20 classes, 964 images) Berkeley segmentation dataset (natural scenes and objects, 300 images)
Baselines: CPMC [Carreira and Sminchisescu, PAMI 2012] Object proposals [Endres and Hoiem, ECCV 2012] gPb+owt+ucm [Arbelaez et al., PAMI 2011]
Evaluation metric:Best covering score w.r.t # of segments
…
Ground truth
0.92
0.75
0.71
Best covering score: 0.92
Slide18
Segmentation quality
ApproachCovering (%)Num of segmentsShape sharing (Ours)84.31448CPMC [Carreira and Sminchisescu]81.61759Object proposals [Endres and Hoiem]81.71540gPb-owt-ucm [Arbelaez et al.]62.81242
PASCAL 2010 dataset
ApproachCovering (%)Num of segmentsShape sharing (Ours)75.61449CPMC [Carreira and Sminchisescu]74.11677Object proposals [Endres and Hoiem]72.31275gPb-owt-ucm [Arbelaez et al.]61.61483
Berkeley segmentation dataset
*Exemplars = PASCAL
Slide19
When does shape sharing help most?
Gain as a function of color easiness and object size
Easy to segment by color
Hard to segment by color
Compared to CPMC [
Carreira
and Sminchisescu., PAMI 2012]
Slide20
Which classes share shapes?
Animals
Vehicles
Semantically disparate
Unexpected
pose variations
Slide21
Shape sharing (ours)
CPMC (Carreira and Sminchisescu)
0.889
0.859
0.903
0.935
0.599
0.638
0.630
0.694
Objects with diverse colors
Example results (
good
)
Slide22
Shape sharing (ours)
CPMC (Carreira and Sminchisescu)
Objects confused by surrounding colors
Example results (good)
0.966
0.875
0.999
0.928
0.508
0.533
0.526
0.685
Slide23
Shape sharing (ours)
CPMC (Carreira and Sminchisescu)
Example results (failure cases)
0.220
0.199
0.713
0.406
0.818
0.934
0.973
0.799
Slide24
Shape sharing: highlights
Non-parametric transfer of shapes across categoriesPartial shape agreement from multiple exemplarsMultiple hypothesis approachMost impact for heterogeneous objects
Code
is available:
http://vision.cs.utexas.edu/projects/
shapesharing
Top-down shape prior in a category-independent way
Download this presentation
DownloadNote - The PPT/PDF document "Shape Sharing for" is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Presentations text content in Shape Sharing for
Shape Sharing for Object Segmentation
Jaechul Kim and Kristen GraumanUniversity of Texas at Austin
Slide2Problem statement
Category-independent object segmentation: Generate object segments in the image regardless of their categories.
Slide3Related work
Top-down, category-specific approache.g., Active Contours (IJCV 1987), Borenstein and Ullman (ECCV 2002), Levin and Weiss (ECCV 2006), Kumar et.al. (CVPR 2005)
Use generic knowledge on object shapese.g., Levinshtein et.al. (ICCV 2009)
Category-independent multiple segmentationse.g., Malisiewicz and Efros (BMVC 2007), Carreira and Sminchisescu (CVPR 2010), Endres and Hoiem (ECCV 2010)
Learn local shapese.g., Ren et.al. (ECCV 2006), Opelt et.al. (CVPR 2006)
Slide4Spectrum of existing approaches
Class-specific
Bottom-up
+ coherent mid-level regions
+ applicable to any image
- prone to over/under-segment
+ robustness to low-level cues- typically viewpoint specific- requires class knowledge!
How to model
top-down shape in a category-independentway?
horse shape priors
color, textures, edges…
Malisiewicz
and
Efros
(BMVC 2007),
Arbelaez et al. (CVPR 2009)Carreira and Sminchisescu (CVPR 2010)Endres and Hoiem (ECCV 2010)
Active Contours (IJCV 1987)Borenstein and Ullman (ECCV 2002) Levin and Weiss (ECCV 2006)Kumar et.al. (CVPR 2005)
e.g.,
e.g.,
Slide5Our goal
Segment even unfamiliar objects with category-independent top-down cues
Top-down segmentation with shape prior
We don’t want to care what is in the image.
Cow? Sheep?
Slide6Our idea:Shape sharing
Semantically close
Semantically disparate
Object shapes are shared among different categories.
Shapes from one class can be used to segment another (
possibly unknown
) class:
Enable category-independent shape priors
Slide7Basis of approach:
transfer through matching
Transfer category-independent shape prior
Exemplar image
Test image
Partial shape
match
Global shape
projection
ground truth
object boundaries
Slide81. Shape projection
via local shape matches
…
Exemplars
Test image
Approach: Overview
+
Shape prior
Color model
Segmentation model per each group
2. Aggregating
the shape projections
Graph-cut
Segmentation hypotheses
3. Multiple figure-ground segmentations
with shape prior
Slide9Approach: Shape projection
Test image
Exemplars
Projection
Aggregation
Segmentation
Vs.
BPLRs
Superpixels
Boundary-Preserving Local Regions (BPLR):
Distinctively shaped
Dense
Repeatable
[Kim & Grauman, CVPR 2011]
Slide10Approach: Shape projection
Test image
Exemplars
…
…
Shape projections via
similarity transform of BPLR matches
Projection
Aggregation
Segmentation
Matched
Exemplar 1
Matched
Exemplar 2
Shape hypotheses
Slide11Approach: Refinement of projections
Refined shape
Initial projection
Exemplar
jigsaw
Projection
Aggregation
Segmentation
Include
superpixels
where majority of pixels overlap projection
Align with bottom-up evidence
Slide12Approach: Aggregating projections
Projection
Aggregation
Segmentation
…
Exploit
partial
agreement
from multiple exemplars
Grouping based on overlap
Slide13+
Shape prior
Color model
Segmentation model per
each
group
Approach: Segmentation
Projection
Aggregation
Segmentation
Figure-ground segmentation using graph-cut
Slide14n-links
Projection
Aggregation
Segmentation
Approach: Graph-cut
data term
smoothness term
Define a graph over image pixels:
node = pixel
edge = cost of a cut between pixels
Energy function to minimize:
Slide15Projection
Aggregation
Segmentation
Approach:
Segmentation
Color likelihood
Fg
Bg
NA
Fg
color histogram
Bg
color histogram
Shape likelihood
Graph-cut optimization
Data term
+
Smoothness term
Slide16Projection
Aggregation
Segmentation
Approach: Multiple segmentations
Parameter controlling
data term bias
Compute multiple segmentations
by varying
foreground bias:
…
Output:
Carreira
and
Sminchisescu
,
CPMC: Automatic Object Segmentation Using Constrained Parametric Min-Cuts
PAMI
2012
.
Slide17Experiments
Exemplar database: PASCAL 2010 segmentation task training set (20 classes, 2075 objects)
Test datasets: PASCAL 2010 segmentation task validation set (20 classes, 964 images) Berkeley segmentation dataset (natural scenes and objects, 300 images)
Baselines: CPMC [Carreira and Sminchisescu, PAMI 2012] Object proposals [Endres and Hoiem, ECCV 2012] gPb+owt+ucm [Arbelaez et al., PAMI 2011]
Evaluation metric:Best covering score w.r.t # of segments
…
Ground truth
0.92
0.75
0.71
Best covering score: 0.92
Slide18Segmentation quality
ApproachCovering (%)Num of segmentsShape sharing (Ours)84.31448CPMC [Carreira and Sminchisescu]81.61759Object proposals [Endres and Hoiem]81.71540gPb-owt-ucm [Arbelaez et al.]62.81242
PASCAL 2010 dataset
ApproachCovering (%)Num of segmentsShape sharing (Ours)75.61449CPMC [Carreira and Sminchisescu]74.11677Object proposals [Endres and Hoiem]72.31275gPb-owt-ucm [Arbelaez et al.]61.61483
Berkeley segmentation dataset
*Exemplars = PASCAL
Slide19When does shape sharing help most?
Gain as a function of color easiness and object size
Easy to segment by color
Hard to segment by color
Compared to CPMC [
Carreira
and Sminchisescu., PAMI 2012]
Slide20Which classes share shapes?
Animals
Vehicles
Semantically disparate
Unexpected
pose variations
Slide21Shape sharing (ours)
CPMC (Carreira and Sminchisescu)
0.889
0.859
0.903
0.935
0.599
0.638
0.630
0.694
Objects with diverse colors
Example results (
good
)
Slide22Shape sharing (ours)
CPMC (Carreira and Sminchisescu)
Objects confused by surrounding colors
Example results (good)
0.966
0.875
0.999
0.928
0.508
0.533
0.526
0.685
Slide23Shape sharing (ours)
CPMC (Carreira and Sminchisescu)
Example results (failure cases)
0.220
0.199
0.713
0.406
0.818
0.934
0.973
0.799
Slide24Shape sharing: highlights
Non-parametric transfer of shapes across categoriesPartial shape agreement from multiple exemplarsMultiple hypothesis approachMost impact for heterogeneous objects
Code
is available:
http://vision.cs.utexas.edu/projects/
shapesharing
Top-down shape prior in a category-independent way
Slide25Approach: Refinement and pruning
Initial projection
Refined shape
Exemplar
jigsaw
Pruned out
Exemplar
Projection
Aggregation
Segmentation
Slide26Segmentation quality
Quality of initial shape projections
Initial shape prior
Exemplar
Approach
Covering (%)
Num
of
segmentsExemplar-based merge (Ours)77.0607Neighbor merge [1]72.25005Bottom-up segmentation [2]62.81242
[
1]
Malisiewicz
and
efros
, BMVC 2007.
[2]
Arbelaez
et.al., PAMI 2011.
Slide27Impact of shapes
CPMC, PASCAL
CPMC, BSD
Object proposal, PASCAL
Object proposal, BSD
Shape sharing’s gain in recall as a function of overlap
Slide28Category-independent vs. dependent
ApproachCovering (%)Category-specific84.7Category-independent (default)84.3Strictly category-independent83.9CPMC81.6Object proposals81.7
Comparison of category-independent shape prior and category-specific variants