Object Segmentation Jaechul Kim and Kristen Grauman University of Texas at Austin Problem statement Categoryindependent object segmentation Generate object segments in the image regardless of their categories ID: 142582
Download Presentation The PPT/PDF document "Shape Sharing for" is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
Shape Sharing for Object Segmentation
Jaechul Kim and Kristen GraumanUniversity of Texas at AustinSlide2
Problem statement
Category-independent object segmentation: Generate object segments in the image regardless of their categories.Slide3
Related work
Top-down, category-specific approache.g., Active Contours (IJCV 1987), Borenstein and Ullman (ECCV 2002), Levin and Weiss (ECCV 2006), Kumar et.al. (CVPR 2005)
Use generic knowledge on object shapes
e.g.,
Levinshtein
et.al. (ICCV 2009)
Category-independent multiple segmentations
e.g.,
Malisiewicz
and
Efros
(BMVC 2007), Carreira and Sminchisescu (CVPR 2010), Endres and Hoiem (ECCV 2010)
Learn local shapese.g., Ren et.al. (ECCV 2006), Opelt et.al. (CVPR 2006)Slide4
Spectrum of existing approaches
Class-specific
Bottom-up
+ coherent mid-level regions
+ applicable to any image
- prone to over/under-segment
+ robustness to low-level cues
- typically viewpoint specific
- requires class knowledge!
How to model
top-down
shape in a
category-independent
way?
horse shape priors
color, textures, edges…
Malisiewicz
and
Efros
(BMVC 2007),
Arbelaez
et al. (CVPR 2009)
Carreira
and
Sminchisescu
(CVPR 2010)
Endres
and
Hoiem
(ECCV 2010)
Active Contours (IJCV 1987)
Borenstein
and
Ullman
(ECCV 2002)
Levin and Weiss (ECCV 2006)
Kumar et.al. (CVPR 2005)
e.g.,
e.g., Slide5
Our goal
Segment even unfamiliar objects with category-independent top-down cues
Top-down segmentation with shape prior
We don’t want to care what is in the image.
Cow? Sheep?Slide6
Our idea:Shape sharing
Semantically close
Semantically disparate
Object shapes are shared among different categories.
Shapes from one class can be used to segment another (
possibly unknown
) class:
Enable category-independent shape priorsSlide7
Basis of approach:
transfer through matching
Transfer category-independent
shape
prior
Exemplar
image
Test image
Partial shape
match
Global shape
projection
ground truth
object boundariesSlide8
1. Shape projection
via local shape matches
…
Exemplars
Test image
Approach: Overview
+
Shape prior
Color model
Segmentation model per each group
2. Aggregating
the shape projections
Graph-cut
Segmentation hypotheses
3. Multiple figure-ground segmentations
with shape priorSlide9
Approach: Shape projection
Test image
Exemplars
Projection
Aggregation
Segmentation
Vs.
BPLRs
Superpixels
Boundary-Preserving Local Regions (BPLR):
Distinctively shaped
Dense
Repeatable
[Kim & Grauman, CVPR 2011]Slide10
Approach: Shape projection
Test image
Exemplars
…
…
Shape projections via
similarity transform of BPLR matches
Projection
Aggregation
Segmentation
Matched
Exemplar 1
Matched
Exemplar 2
Shape hypothesesSlide11
Approach: Refinement of projections
Refined shape
Initial projection
Exemplar
jigsaw
Projection
Aggregation
Segmentation
Include
superpixels
where majority of pixels overlap projection
Align with bottom-up evidenceSlide12
Approach: Aggregating projections
Projection
Aggregation
Segmentation
…
Exploit
partial
agreement
from multiple exemplars
Grouping based on overlapSlide13
+
Shape prior
Color model
Segmentation model per
each
group
Approach: Segmentation
Projection
Aggregation
Segmentation
Figure-ground segmentation using graph-cutSlide14
n-links
Projection
Aggregation
Segmentation
Approach: Graph-cut
data term
smoothness term
Define a graph over image pixels:
node = pixel
edge = cost of a cut between pixels
Energy function to minimize:Slide15
Projection
Aggregation
Segmentation
Approach:
Segmentation
Color likelihood
Fg
Bg
NA
Fg
color histogram
Bg
color histogram
Shape likelihood
Graph-cut optimization
Data term
+
Smoothness term Slide16
Projection
Aggregation
Segmentation
Approach: Multiple segmentations
Parameter controlling
data term bias
Compute multiple segmentations
by varying
foreground bias:
…
Output:
Carreira
and
Sminchisescu
,
CPMC: Automatic Object Segmentation Using Constrained Parametric Min-Cuts
PAMI
2012
.Slide17
Experiments
Exemplar database:
PASCAL 2010 segmentation task training set (20 classes, 2075 objects)
Test datasets:
PASCAL 2010 segmentation task validation set (20 classes, 964 images)
Berkeley segmentation dataset (natural scenes and objects, 300 images)
Baselines:
CPMC
[
Carreira
and
Sminchisescu
, PAMI 2012]
Object proposals
[Endres
and Hoiem, ECCV 2012]
gPb+owt+ucm
[
Arbelaez et al., PAMI 2011]
Evaluation metric:Best covering score
w.r.t # of segments
…
Ground truth
0.92
0.75
0.71
Best covering score: 0.92Slide18
Segmentation quality
ApproachCovering (%)
Num
of
segments
Shape sharing (Ours)
84.3
1448
CPMC [
Carreira
and
Sminchisescu]81.6
1759Object proposals [Endres and Hoiem
]
81.71540gPb-owt-ucm
[Arbelaez et al.]
62.81242
PASCAL 2010 dataset
Approach
Covering (%)
Num of segments
Shape sharing (Ours)
75.61449
CPMC [Carreira and Sminchisescu]
74.11677
Object
proposals [
Endres
and
Hoiem
]
72.3
1275
gPb-owt-ucm
[
Arbelaez
et al.]
61.6
1483
Berkeley segmentation dataset
*Exemplars = PASCALSlide19
When does shape sharing help most?
Gain as a function of color easiness and
object size
Easy to segment by color
Hard to segment by color
Compared to CPMC [
Carreira
and
Sminchisescu
., PAMI
2012]Slide20
Which classes share shapes?
Animals
Vehicles
Semantically disparate
Unexpected
pose variations Slide21
Shape sharing (ours)
CPMC (Carreira and
Sminchisescu
)
0.889
0.859
0.903
0.935
0.599
0.638
0.630
0.694
Objects with diverse colors
Example results (
good
)Slide22
Shape sharing (ours)
CPMC (Carreira and
Sminchisescu
)
Objects confused by surrounding colors
Example results (
good
)
0.966
0.875
0.999
0.928
0.508
0.533
0.526
0.685Slide23
Shape sharing (ours)
CPMC (Carreira and
Sminchisescu
)
Example results (
failure cases
)
0.220
0.199
0.713
0.406
0.818
0.934
0.973
0.799Slide24
Shape sharing: highlights
Non-parametric transfer of shapes across categoriesPartial shape agreement from multiple exemplarsMultiple hypothesis approachMost impact for heterogeneous objects
Code
is available:
http://vision.cs.utexas.edu/projects/
shapesharing
Top-down shape prior in a category-independent waySlide25
Approach: Refinement and pruning
Initial projection
Refined shape
Exemplar
jigsaw
Pruned out
Exemplar
Projection
Aggregation
SegmentationSlide26
Segmentation quality
Quality of initial shape projections
Initial shape prior
Exemplar
Approach
Covering (%)
Num
of
segments
Exemplar-based merge (Ours)
77.0
607
Neighbor
merge [1]
72.2
5005
Bottom-up segmentation [2]
62.8
1242
[
1]
Malisiewicz
and
efros
, BMVC 2007.[2]
Arbelaez
et.al., PAMI 2011. Slide27
Impact of shapes
CPMC, PASCAL
CPMC, BSD
Object proposal, PASCAL
Object proposal, BSD
Shape sharing’s gain in recall as a function of overlapSlide28
Category-independent
vs. dependent
Approach
Covering (%)
Category-specific
84.7
Category-independent
(default)
84.3
Strictly category-independent
83.9
CPMC
81.6
Object proposals
81.7
Comparison of category-independent shape prior and category-specific variants