Liyan Zhang, Dmitri V. Kalashnikov, - PowerPoint Presentation

Slide1

Liyan Zhang, Dmitri V. Kalashnikov, Sharad MehrotraDepartment of Computer ScienceUniversity of California, Irvine

A Unified Framework for Context Assisted Face ClusteringSlide2

Introduction

Explosion of Media Data

Human is Center

Face Clustering

Face Tagging

User FeedbackSlide3

OutlineIntroduction to Face ClusteringTraditional Approaches for Face Clustering

The Proposed Context Assisted FrameworkExperimental Results

Conclusions and Future WorkSlide4

Face Appearance based Approach

Facial Features

Face Similarity Graph

Clustering Algorithm

Detected Faces

……

Clustering ResultsSlide5

Appearance based Face Clustering Results

Good Clustering Results

H

igh Precision,High Recall

Tight Clustering Threshold

H

igh Precision, Low Recall

loose Clustering Threshold

Low

Precision, High Recall

Too Much Merging Work!Slide6

Drawbacks of Facial Similarities

Same People Look

Different

Different Pose

Different Expression

Different Illumination

Different Occlusion

Different

People Look

The Same

Boy

Girl

Boy

GirlSlide7

Context Information Helps

Common Scene:

Geo Location

Captured Time

Image Background

Social Context:

People Co-occur

Human Attributes:

Age

Ethnicity

Gender

Hair

…

Clothing:

Cloth colorSlide8

Related Work

[1] Y. J. Lee and K. Grauman

. Face discovery with social context. In BMVC, 2011.

[3] N. Kumar and et al. Describable visual attributes for face verification and image search. In IEEE TPAMI, 2011.

[2] A. Gallagher and T. Chen. Clothing

cosegmentation

for recognizing people. In IEEE CVPR, 2008.

People Co-occurrence

[1]

Clothing

[2]

Human Attributes

[3]

Heterogeneous Context Feature

Single

Context Type

Face Level

Cluster Level

Context

Prior work

Context

Heterogeneous

Single

Slide9

The Framework

Photo Collection

Detected Faces

……

Initial Clusters : High Precision, Low Recall

……

Iterative Merging

cont

Common Scene

People Co-occurrence

Human Attributes

Clothing

…

……

Final Clusters: High Precision, High RecallSlide10

Context Features Extraction

Context Similarities

Context Constraints

Integrate

Same?

Diff ?Slide11

Common Scene

Image captured time, camera model, image visual features

Common

Scene

Same

people

I

1

I

2

I

3

I

1

I

2

C

1

C

2Slide12

People Co-occurrence

same

diff

diffSlide13

People Co-occurrence Cluster Co-Occurrence Graph

1

1

1

1

1

1

I

2

I

1

1

1

1

1

1

f

1

f

2

f

3

f

4

f

5

f

6

f

7

f

8Slide14

Human Attributes

same

diff

73-D

73-D

N. Kumar and et al. Describable visual attributes for face verification and image search. In IEEE TPAMI, 2011.Slide15

Human Attributes

Attribute

Attribute

Attribute

C5

f

1

f

2

f

3

Attribute

Attribute

Attribute

Similar?

C5

cosine

Only One Child

Many Children!

AGE attribute

Bootstrapping:

Learn Weights From Dataset!

Different Attributes

Different WeightsSlide16

Human AttributesFaceAttributes

Label

C

1

C

1

C

1

~ C

1

~ C

1

~ C1~ C

1

~ C1

Attribute

Attribute

Attribute

C5

f

1

f

2

f

3

Attribute

Attribute

Attribute

f

4

f

5

f

6

f

7

f

8

diff

diff

diff

diff

diff

Attribute

Attribute

Attribute

Attribute

Attribute

Train

Classifier

C

1

? Slide17

Clothing

Similarity from clothes

Cloth color

hist

similarity

Time diff

Time slot threshold

Time Sensitive!

Diff

Same

Time diff << S

Time

diff >>

SSlide18

Context Features

Cluster-Level

Context Similarities

Common Scene

People Co-occurrence

Human Attributes

ClothingSlide19

Context Features

Cluster-Level

Context Constraints

Diff

Co-occurred people

Diff

Distinct Attributes

Time & Location

Time: t s Indoor

Time: (t+1)s Outdoor

DiffSlide20

Single Context Feature Fails

Common Scene

People Co-occurrence

Diff

Same

Different Attributes

Different Clothing

Co-occurred People

Diff

SameSlide21

Integration is Required

Aggregation?

Set a rule?

Context Similarities

Context Constraints

Integrate

?

Context Features

Merge

Merge

Not

Y=a +b +c +d +e +…

The importance of features differ with different dataset!

Learn Rules from Each Dataset!Slide22

How to Learn Rules?

Photo Collection

Split Initial

clusters

Same Pairs

Manually Label

Learning

Rules

Apply Rules

Training Dataset

……

Initial Clusters : High Precision, Low Recall

Facial Features

cont

Common Scene

People Co-occurrence

Human Attributes

Clothing

…

Context Similarity

& Constraint

Learn from data

Itself!

Apply rules

Learning

Rules

Training Dataset

Automatic Label

Context Constraints

Diff Pairs

BootstrappingSlide23

pairsLabelSame

Same

Same

Same

Same

Diff

Diff

Diff

Diff

Cost-sensitive

DTC

Splitting

Training

Diff

Example of Automatic Labeling

same

same

diff

same

diff

diff

sameSlide24

pairsLabelSame

Same

Same

Same

Same

Diff

Diff

Diff

Diff

Cost-sensitive

DTC

pairs

predict

Same

Same

Same

Same

Same

Diff

Same

Diff

…

…

(

):

5 same

(

):

1 same

Splitting

Training

Predicting

Diff

1

st

Splitting—Training--PredictingSlide25

pairsLabelSame

Same

Same

Same

Same

Diff

Diff

Diff

Diff

Cost-sensitive

DTC

pairs

predict

Same

Same

Same

Same

Diff

Diff

Diff

Diff

…

…

(

):

4 same

(

):

0 same

Splitting

Training

Predicting

Diff

2

nd

Splitting—Training--PredictingSlide26

Combine results

C1-C3:

5 same

C2-C3:

1 same

C1-C3:

4 same

C2-C3:

0 same

C1-C3:

9 same

C2-C3:

1 same

Merge

C1-C3

pairs

predict

Same

Same

Same

Same

Same

Diff

Same

Diff

…

…

pairs

predict

Same

Same

Same

Same

Diff

Diff

Diff

Diff

…

…

1

st

Time

2

nd

TimeSlide27

Unified Framework

Pure

clusters

splitting

training

prediction

splitting

training

prediction

splitting

training

prediction

…

Final

Decision

Extracted

Faces

Merge

Pairs?

YES

No

Results

Iterative Merging

Photo

Album

Facial

Context

FeaturesSlide28

Experiment Datasets

Gallagher

Wedding

SurveillanceSlide29

Evaluation Metrics

B-cubed Precision

and RecallSlide30

Performance Comparison

Facial

Features

Photo

Album

Context

Features

Pure

Clusters

Splitting

Training Predicting

Process

Merge

Decision

Update

Our Approach:

Precision

Recall

Picasa:

Cluster Threshold

50



95

Different

Clusters

Affinity

Propagation:

Context Similarities Aggregation

Facial Similarities

Different Parameter: p

Different

ClustersSlide31

ResultsSlide32

Results

High Precision

Higher RecallSlide33

Results

High Precision,

662 clusters

31 Real Person,

631 Merging

High Precision,

203 clusters

31 Real Person,

172 Merging

4 TimesSlide34

Results

Less Clusters

Less Manual MergingSlide35

ResultsSlide36

Conclusion and Future Work

Heterogeneous

Context Features

Context

Constraint

Co-occur

People

Distinct

Attributes

Time &

Space

Context

Similarity

Common

Scene

People

Co-occur

Human

Attributes

Clothing

Single

Context

Feature

Context

Similarity

Prior work

Our Approach

Efficiency?

User Feedback?

Break points for precision dropping?

Future work

Bootstrapping

Integration

Iterative

Merging

High precision

High recallSlide37

Thank you!Questions?