Collections through Contextual Focal Points Kai Xu Rui Ma Hao Zhang Chenyang Zhu Ariel Shamir Daniel CohenOr Hui Huang Shenzhen VisuCA Key Lab ID: 920671
Download Presentation The PPT/PDF document "Organizing Heterogeneous Scene" is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
Slide2Organizing Heterogeneous Scene
Collections through
Contextual Focal Points
Kai Xu, Rui Ma, Hao Zhang, Chenyang Zhu,Ariel Shamir, Daniel Cohen-Or, Hui Huang
Shenzhen VisuCA Key Lab / SIATSimon Fraser UniversityNational University of Defense TechnologyThe Interdisciplinary CenterTel Aviv University
Slide3Rapid growth of 3D data
Trimble 3D Warehouse
over
2 million modelsOrganizing large 3D datasets
3
Slide4Organization
Examples
Categorization tree
in ImageNet[Stanford]
Categorization trees of 3D shape set[Huang et al. 2013]
4
Slide5The key to organization
Grouping & relating similar contents
How to compare complex things?5
For heterogeneous 3D shapes …qualitative analysis
[Huang et al. 2013]
Slide6Objects vs. Scenes
heterogeneous
hybrid
bedroom
d
ining room
living room
office
studio
Slide7London
Paris
Comparing complex things
7
Slide8London
Paris
New York City
Milan
Comparing complex things
8
Slide9London
Paris
Berlin
RomeMadrid
Comparing complex things
9
Slide10The key point
Comparing complex things:
focal pointsDetermining focal points: within a context
10
Slide11How to measure similarity of scenes?
Especially hard for hybrid scenes
?
?
distance
11
Slide12How to measure similarity of scenes
3D scenes should be compared
wrt
a focal point
focal : a
subscene
12
Slide13How to measure similarity of scenes
3D scenes should be
compared wrt a focal point
13
Slide14How to determine focal points?
Contextual analysis: co-analyzing a set
Characterize a semantic scene typeFrequently appear in a category
… …
l
iving room
14
Slide15How to determine focal points?
Frequency alone is not enough …
Trivially frequent subscene is meaningless
…
a single
chair may appear
in
most indoor
scenes
15
Slide16How to determine focal points?
Discriminant
Frequent only within the set of semantically related scenesCharacterize the scene category
…
…
bedroom
l
iving room
unknown
unknown
16
Slide17How to determine focal
points?
Focal point detection relies on clusteringFrequent only within some clusterScene clustering is guided by focal pointsCharacterized by representative focal points
Coupled problems …17
Slide18Overview
Basic rep.: structural graphs
Interleaving optimization
18
Slide19Overview
Interleaving optimization
Focal point detection
Focal-based clustering
19
Slide20Overview
Focal-based organization
20
Slide21Focal-based scene collection exploration
21
Slide22Main ideas
How to measure similarity of complex things?
Based on focal pointsHow to determine focal points?Co-analysis of a collectionCoupled with clusteringFocal points provide a way to relate complex things
Slide23Outline
Datasets
Structural graph & layout similarityFocal-driven scene co-analysisThe objectiveInterleaving optimization23
Slide24Outline
Datasets
Structural graph & layout similarityFocal-driven scene co-analysisThe objective Interleaving optimization
24
Slide25Dataset
Collection
#Scenes
#Objects
#Scene
categories
#Hybrid
scenesStanford[Fisher et al. 2012]1323,46150
Tsinghua
[Xu et al. 2013
]
792
13,365
6
102
25
Slide26Outline
Datasets
Structural graph& layout similarityFocal-driven scene co-analysisThe objectiveInterleaving optimization
26
Slide27Structural graph
Nodes: individual objects
Edge: relationships (support, proximity
) 27
Slide28Layout similarity
Measuring edge similarity
Spatial arrangement of OBBs, not edge tags
28
Slide29Outline
Datasets
Structural graph & layout similarityFocal-driven scene co-analysisThe objectiveInterleaving optimization
29
Slide30Objective
Our core problem: clustering
Objective: overall compactness of clusters
Per-cluster compactness: focal-centric scene similarity
Optimize clustering
and focal points
30
Slide31Interleaving optimization
Focal extraction: cluster-guided graph mining
Fix clustering, optimize focal pointsScene clustering: focal-induced scene clusteringFix focals, optimize clustering
31
Slide32Interleaving optimization
Focal extraction: cluster-guided graph mining
Fix clustering, optimize focal pointsScene clustering: focal-induced scene clusteringFix focals
, optimize clustering32
Slide33Focal extraction:
graph mining
……
……
……
……
33
Slide34Focal extraction:
graph mining
……
……
……
……
34
Structural graph
Slide35Focal extraction:
graph mining
Frequent pattern mining
……
……
……
……
35
Slide36Focal extraction:
graph mining
Frequent pattern mining……
………………
36
Slide37Focal extraction:
graph mining
…………
…………
37
Focal embedding
Slide38Focal extraction
: cluster-guided
miningTrivially frequent substructuresE.g. a single chairFrequent but not discriminantCluster-guided mining
Mining substructures that characterize a clusterUsing clusters to weight frequency38
Slide39Cluster-guided
mining: illustrative example
Total # of occurrence:
#
39
#
#
Frequent pattern
mining result:
Cluster-guided
mining: illustrative example
40
Using clusters to weight frequency
cluster 1
cluster 2
cluster 3
Slide41Cluster-guided
mining: illustrative example
cluster 1
cluster 2
cluster 3
is not discriminant
41
Slide42Cluster-guided
mining: illustrative example
cluster 1
cluster 2
cluster 3
is representative focal of cluster 2
42
Slide43Cluster-guided
mining: illustrative example
cluster 1
cluster 2
cluster 3
43
Cluster-guided mining result:
Slide44Interleaving optimization
Focal extraction: cluster-guided graph mining
Fix clustering, optimize focal pointsScene clustering: focal-induced scene clusteringFix focals, optimize clustering
44
Slide45Scene clustering
Representation: Bag-of-Word
(
BoW
)
feature
45
Slide46Scene clustering
Subspace clustering [Wang et al. 2011]
46
Slide47Scene clustering
Indicator feature: not very informative
Really similar?
47
Slide48FCGK
Per-cluster compactness defined by
focal-centric graph kernel
(FCGK):Focal-induced scene clustering
Scaling factor
Root walk kernel
[Fisher et al. 2011]
48
Slide49Focal-induced scene clustering
R
e
weight Bag-of-Word feature
Comp.: 0.2
Comp.: 1.0
Overall Comp.: 1.2
49
Slide50Focal-induced scene clustering
Reweighted subspace
clustering
Comp.: 1.0
Comp.: 1.8
Overall Comp.: 2.8
50
Slide51Interleaving optimization
Objective: maximize overall compactness
51
Slide52Post-processing
Cluster attachment
Focal joining
Cluster overlapLarge-scale and non-local
focals52
Slide53Co-analyzing scene collection
53
Slide5454
Slide55Results
Time
focal point extraction (~5%)scene clustering (~90% with a Matlab implementation)
CollectionTime (min) #Focal#Non local focal%Multi-focal scenes
Stanford (132)3.224450.4%
Tsinghua (792)10.5347
46.1%55
Slide56Compactness Plot
Change of overall compactness while interleaving optimization
Stanford
Tsinghua56
Slide57Detected focal points (Tsinghua)
……
Non local
focals
57
Slide58Results – Comparing to graph kernel [Fisher et al. 2011]
FCGK is more discriminative;
New perspective to compare complex scenes 58
Slide59Application – Comprehensive retrieval
Query
……
59
Slide60Application
–
scene collection exploration
“gateway
”
60
Slide61Limitations
Use object labels:
Handling of noisy or incomplete labels?Pure geometry analysis?Structural graphs model flat arrangements of objectsHierarchical organization potentially advantageous?
61
Slide62Future work
Apply to other datasets
E.g. large and heterogeneous collections of imagesScene synthesisSubstitute sub-scenes instead of one object at a time62
Slide63Acknowledgement
Anonymous reviewers
Datasets: [Fisher et al. 2012] and [Xu et al. 2013]Research grants:NSFC China, NSERC Canada, National 863 Program of China, Shenzhen Innovation Program, CPSF China,
Israel Science Foundation63
Slide64Thank you!
An
exploratory
path through an overlap smoothly transits between two scene clusters.64