Improving the Recognition of Faces Occluded by Facial Accessories - PowerPoint Presentation

Slide1

Improving the Recognition of Faces Occluded by Facial Accessories

16/03/2011

1

Rui

Min

Multimedia Communications Dept.

EURECOM

Sophia Antipolis, Francemin@eurecom.fr

Abdenour Hadid Machine Vision GroupUniversity of OuluOulu, Finlandhadid@ee.oulu.fi

Jean-Luc DugelayMultimedia Communications Dept.EURECOMSophia Antipolis, Francejld@eurecom.fr

Outline

Research Problem and ObjectivesState of the ArtFramework of the Proposed Approach

Occlusion DetectionFace RecognitionExperiments

Conclusions & Discussions

16/03/2011

2

Research Problem

Facial Occlusions: Sunglasses, Scarf, Medical Mask, Beards etc.Face Recognition in Non-Cooperative Systems (e.g. Video Surveillance)

Security Issues:Football HooligansATM Criminals

16/03/2011

3

Main Goals

Address the Face Recognition under Occlusions caused by Facial AccessoriesIn Particular: Improving the Recognition of Faces Occluded by Sunglasses and Scarf

Specifically take into account the Occlusion AnalysisRobustnessSimplicity

16/03/2011

4

State of the Art

Holistic approaches are not robust to partial occlusions e.g.: PCA, LDA and ICALocal feature-based and component based methods

Less sensitive to occlusions than the holistic methodse.g. : LFA (local feature analysis), LS-ICA (local salient ICA)Other new trends: e.g. Sparse Representation, Partial-SVM

(More Recently) Occlusion Analysis prior to Face

recognition

Observation : Prior knowledge of occlusion significantly improve performance

PCA based approach +manually annotated Occlusion information [1]

S-LNMF (Selective Local Non-negative Matrix Factorization): automatic occlusion detection + LNMF based FR [2]

16/03/2011

5

Our Framework

Two Step Algorithm:Occlusion Detection in Local PatchesFace Recognition based on Local Binary Patterns (LBP)

Recognizing a Probe faceCompute the LBP representationDivide the image into local patches

Occlusion detection in each patch

Non-occluded patches are selected for recognition

16/03/2011

6

Occlusion Detection

Facial Feature based algorithms:The information of facial features (such as mouth or skin color) is exploited to decide whether or not a face is occluded.

Weakness: Structure variation of the occluded part might be wrongly categories as mouth/eyes Color variation

of the occluded part might be wrongly recognized as skin color

Learning based algorithms:

A large number of positive (clean faces) and negative (occluded faces) samples to train a classifier, which can predict the label of an unknown face.

Our previous study: <<Robust Scarf Detection prior to Face Recognition>>[3]

Learning based method is more robust against texture variations

Better tolerate image degradation

16/03/2011

7

Occlusion Detection – cont.

Image DivisionFeature Extraction: Gabor Wavelet filteringDimensionality Reduction: Principal Component Analysis (PCA)

Classification: Support Vector Machine (SVM)

16/03/2011

8

Feature extraction

Dimensionality

reduction

SVM-based classification

Feature extraction

Dimensionality

reduction

SVM-based classification

Occlusion Detection – cont.

Gabor Wavelet based Feature Extraction:Image Filtering by Gabor Wavelets

Dimensionality Reduction by PCA

Construct a data set consists of the features extracted from the occluded and non-occluded patches:

Compute the eigenvectors of the covariance matrix of the centered S

Project the extracted features onto the eigenspace

16/03/2011

9

Occlusion Detection – cont.

SVM based Occlusion DetectionOcclusion Detection is considered as a two-class classification problem

For a training set consisting of N pairs , is the label indicates occlusion or not.SVM finds the maximum-margin hyper-plane to separate the data by:

Kernel in use: Radial Basis Function (RBF)

The implementation is provided by LIBSVM (http://www.csie.ntu.edu.tw/~cjlin/libsvm/)

16/03/2011

10

Face Representation using LBP

16/03/2011

11

Local Binary Patterns

Discriminative power

Computational simplicity

Robustness to monotonic gray scale changes (e.g. illumination variations)

Robust to local deformation, geometric transformation and misalignment to some extent.

Local patch based approach: easily connected with occlusion analysis

Face Representation using LBP-cont.

16/03/2011

12

Feature Extraction

The LBP code for pixel is given by:

The Thresholding function:

Face Representation

Representing the non-occluded facial components

Recognition

Chi-square distance

Nearest-neighbor classifier

Experimental Data

16/03/2011

13

AR

Face Database [4]

A standard testing dataset for occluded face recognition

More than 4000 face images of 126 subjects (70 men and 56 women)

Facial expressions, illumination conditions and occlusions (sunglasses and scarf)

2 sessions: 14 days interval

Experimental Setup

16/03/2011

14

Faces are Cropped

, normalized and down-sampled into 128*128 pixels

LBP operator: (using only uniform patterns, 8 equally spaced pixels on a circle of radius 2)

Face images are divided into 64

blocks (size: 16*16)

Experimental Setup – cont.

16/03/2011

15

Training for the Occlusion Detector

Random selection of 150

non

-occluded faces, 150 faces occluded by

scarf

, 150 faces occluded by

sunglasses

The

upper

parts of the images are used to train the

sunglass detector

The

lower

parts of the images are used to train the

scarf detector

Result of Occlusion

Detection

Results – Experiment 1

16/03/2011

16

Experiment

1 – Justification of the 2-steps approach

Gallery

images: 240

non-occluded

faces from

session 1 (with 3 facial expressions

: neutral, smile and anger)

Evaluation

images: corresponding 240 non-occluded

faces from

session 2

, 240 faces with

scarf

and 240 faces with

sunglasses

from

session 1

Occluded faces are with three different illumination variations

Algorithms in Comparison:

PCA [5

], FA-PCA and

LBP [6

]

FA-PCA: Occlusion Detection Prior to PCA based Face Recognition

Results – Experiment 1 – cont.

16/03/2011

17

Results

Results – Experiment 2

16/03/2011

18

Experiment

2 –

Comparison with the state of the art approach

S-LNMF (

Selective Local Non-negative Matrix Factorization

) [2]

Gallery

images: 240

non-occluded faces from

session 1

Evaluation

images: 240 faces with

scarf

and 240

faces

with sunglasses from

session 2

80 faces with extreme facial expression (

scream

) from

session 2.

80 faces with illumination variations (

Right-Light

) from

session 2

.

Results

Conclusions

19

A novel approach for improving the recognition of occluded faces is proposedState-of-the art in face recognition under occlusion is reviewedA new approach to scarf and sunglasses detection is thoroughly described.

Extensive experimental analysis is conducted, demonstrating significant performance enhancement using the proposed approach compared to many other methods under various configurations

16/03/2011

20

Future Works

Address face recognition under general occlusions (sunglasses, scarves, beards, long hairs, caps, extreme facial make-ups etc.)Automatic face detection under severe occlusions

Automatic face recognition robust to occlusion in video surveillance

16/03/2011

21

References

[1] A. Rama, F. Tarres, L. Goldmann, and T.

Sikora

, "More robust face recognition by considering occlusion information," Automatic Face & Gesture Recognition, 2008. FG '08. 8th IEEE International Conference on , vol., no., pp.1-6, 17-19 Sept. 2008.

[2] H. J. Oh, K. M. Lee, and S. U. Lee, “Occlusion invariant face recognition using selective local non-negative matrix factorization basis images,” Image Vision

Comput

., vol. 26, no. 11, pp. 1515-1523, Nov. 2008.[3] R. Min, A. D'angelo, J.-L. Dugelay

, “Efficient scarf detection prior to face recognition,” EUSIPCO 2010, 18th European Signal Processing Conference, pp 259-263, Aug. 2010. [4] A.M. Martinez and R. Benavente, " The AR face database," Technical report, CVC Technical report, no. 24, 1998.[5] M. Turk, and A.

Pentland, 1991. “Eigenfaces for recognition,” J. Cognitive Neuroscience, vol. 3, no. 1, pp. 71-86, Jan. 1991.[6] T . Ahonen, A.

Hadid

, and M. Pietikäinen

, “Face recognition with local binary patterns,” Computer Vision, ECCV 2004 Proceedings, Lecture Notes in Computer Science 3021, Springer, 469-481.

16/03/2011

Thank you!

16/03/2011 - - p 22