Sharad Mehrotra Department of Computer Science University of California Irvine A Unified Framework for Context Assisted Face Clustering Introduction Explosion of Media Data Human is Center ID: 473863
Download Presentation The PPT/PDF document "Liyan Zhang, Dmitri V. Kalashnikov," is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
Liyan Zhang, Dmitri V. Kalashnikov, Sharad MehrotraDepartment of Computer ScienceUniversity of California, Irvine
A Unified Framework for Context Assisted Face ClusteringSlide2
Introduction
Explosion of Media Data
Human is Center
Face Clustering
Face Tagging
User FeedbackSlide3
OutlineIntroduction to Face ClusteringTraditional Approaches for Face Clustering
The Proposed Context Assisted FrameworkExperimental Results
Conclusions and Future WorkSlide4
Face Appearance based Approach
Facial Features
Face Similarity Graph
Clustering Algorithm
Detected Faces
……
Clustering ResultsSlide5
Appearance based Face Clustering Results
Good Clustering Results
H
igh Precision,High Recall
Tight Clustering Threshold
H
igh Precision, Low Recall
loose Clustering Threshold
Low
Precision, High Recall
Too Much Merging Work!Slide6
Drawbacks of Facial Similarities
Same People Look
Different
Different Pose
Different Expression
Different Illumination
Different Occlusion
Different
People Look
The Same
Boy
Girl
Boy
GirlSlide7
Context Information Helps
Common Scene:
Geo Location
Captured Time
Image Background
Social Context:
People Co-occur
Human Attributes:
Age
Ethnicity
Gender
Hair
…
Clothing:
Cloth colorSlide8
Related Work
[1] Y. J. Lee and K. Grauman
. Face discovery with social context. In BMVC, 2011.
[3] N. Kumar and et al. Describable visual attributes for face verification and image search. In IEEE TPAMI, 2011.
[2] A. Gallagher and T. Chen. Clothing
cosegmentation
for recognizing people. In IEEE CVPR, 2008.
People Co-occurrence
[1]
Clothing
[2]
Human Attributes
[3]
Heterogeneous Context Feature
Single
Context Type
Face Level
Cluster Level
Context
Prior work
Context
Heterogeneous
Single
Slide9
The Framework
Photo Collection
Detected Faces
……
Initial Clusters : High Precision, Low Recall
……
Iterative Merging
cont
Common Scene
People Co-occurrence
Human Attributes
Clothing
…
……
Final Clusters: High Precision, High RecallSlide10
Context Features Extraction
Context Similarities
Context Constraints
Integrate
Same?
Diff ?Slide11
Common Scene
Image captured time, camera model, image visual features
Common
Scene
Same
people
I
1
I
2
I
3
I
1
I
2
C
1
C
2Slide12
People Co-occurrence
same
diff
diffSlide13
People Co-occurrence Cluster Co-Occurrence Graph
1
1
1
1
1
1
I
2
I
1
1
1
1
1
1
f
1
f
2
f
3
f
4
f
5
f
6
f
7
f
8Slide14
Human Attributes
same
diff
73-D
73-D
N. Kumar and et al. Describable visual attributes for face verification and image search. In IEEE TPAMI, 2011.Slide15
Human Attributes
Attribute
Attribute
Attribute
C5
f
1
f
2
f
3
Attribute
Attribute
Attribute
Similar?
C5
cosine
Only One Child
Many Children!
AGE attribute
Bootstrapping:
Learn Weights From Dataset!
Different Attributes
Different WeightsSlide16
Human AttributesFaceAttributes
Label
C
1
C
1
C
1
~ C
1
~ C
1
~ C1~ C
1
~ C1
Attribute
Attribute
Attribute
C5
f
1
f
2
f
3
Attribute
Attribute
Attribute
f
4
f
5
f
6
f
7
f
8
diff
diff
diff
diff
diff
Attribute
Attribute
Attribute
Attribute
Attribute
Train
Classifier
C
1
? Slide17
Clothing
Similarity from clothes
Cloth color
hist
similarity
Time diff
Time slot threshold
Time Sensitive!
Diff
Same
Time diff << S
Time
diff >>
SSlide18
Context Features
Cluster-Level
Context Similarities
Common Scene
People Co-occurrence
Human Attributes
ClothingSlide19
Context Features
Cluster-Level
Context Constraints
Diff
Co-occurred people
Diff
Distinct Attributes
Time & Location
Time: t s Indoor
Time: (t+1)s Outdoor
DiffSlide20
Single Context Feature Fails
Common Scene
People Co-occurrence
Diff
Same
Different Attributes
Different Clothing
Co-occurred People
Diff
SameSlide21
Integration is Required
Aggregation?
Set a rule?
Context Similarities
Context Constraints
Integrate
?
Context Features
Merge
Merge
Not
Y=a +b +c +d +e +…
The importance of features differ with different dataset!
Learn Rules from Each Dataset!Slide22
How to Learn Rules?
Photo Collection
Split Initial
clusters
Same Pairs
Manually Label
Learning
Rules
Apply Rules
Training Dataset
……
Initial Clusters : High Precision, Low Recall
Facial Features
cont
Common Scene
People Co-occurrence
Human Attributes
Clothing
…
Context Similarity
& Constraint
Learn from data
Itself!
Apply rules
Learning
Rules
Training Dataset
Automatic Label
Context Constraints
Diff Pairs
BootstrappingSlide23
pairsLabelSame
Same
Same
Same
Same
Diff
Diff
Diff
Diff
Cost-sensitive
DTC
Splitting
Training
Diff
Example of Automatic Labeling
same
same
diff
same
diff
diff
sameSlide24
pairsLabelSame
Same
Same
Same
Same
Diff
Diff
Diff
Diff
Cost-sensitive
DTC
pairs
predict
Same
Same
Same
Same
Same
Diff
Same
Diff
…
…
(
):
5 same
(
):
1 same
Splitting
Training
Predicting
Diff
1
st
Splitting—Training--PredictingSlide25
pairsLabelSame
Same
Same
Same
Same
Diff
Diff
Diff
Diff
Cost-sensitive
DTC
pairs
predict
Same
Same
Same
Same
Diff
Diff
Diff
Diff
…
…
(
):
4 same
(
):
0 same
Splitting
Training
Predicting
Diff
2
nd
Splitting—Training--PredictingSlide26
Combine results
C1-C3:
5 same
C2-C3:
1 same
C1-C3:
4 same
C2-C3:
0 same
C1-C3:
9 same
C2-C3:
1 same
Merge
C1-C3
pairs
predict
Same
Same
Same
Same
Same
Diff
Same
Diff
…
…
pairs
predict
Same
Same
Same
Same
Diff
Diff
Diff
Diff
…
…
1
st
Time
2
nd
TimeSlide27
Unified Framework
Pure
clusters
splitting
training
prediction
splitting
training
prediction
splitting
training
prediction
…
Final
Decision
Extracted
Faces
Merge
Pairs?
YES
No
Results
Iterative Merging
Photo
Album
Facial
Context
FeaturesSlide28
Experiment Datasets
Gallagher
Wedding
SurveillanceSlide29
Evaluation Metrics
B-cubed Precision
and RecallSlide30
Performance Comparison
Facial
Features
Photo
Album
Context
Features
Pure
Clusters
Splitting
Training Predicting
Process
Merge
Decision
Update
Our Approach:
Precision
Recall
Picasa:
Cluster Threshold
50
95
Different
Clusters
Affinity
Propagation:
Context Similarities Aggregation
Facial Similarities
Different Parameter: p
Different
ClustersSlide31
ResultsSlide32
Results
High Precision
Higher RecallSlide33
Results
High Precision,
662 clusters
31 Real Person,
631 Merging
High Precision,
203 clusters
31 Real Person,
172 Merging
4 TimesSlide34
Results
Less Clusters
Less Manual MergingSlide35
ResultsSlide36
Conclusion and Future Work
Heterogeneous
Context Features
Context
Constraint
Co-occur
People
Distinct
Attributes
Time &
Space
Context
Similarity
Common
Scene
People
Co-occur
Human
Attributes
Clothing
Single
Context
Feature
Context
Similarity
Prior work
Our Approach
Efficiency?
User Feedback?
Break points for precision dropping?
Future work
Bootstrapping
Integration
Iterative
Merging
High precision
High recallSlide37
Thank you!Questions?