Face Recognition - PowerPoint Presentation

Slide1

Face Recognition

Image Understanding

Xuejin ChenSlide2

Face Recogntion

Good websites

http://www.face-rec.org/

Eigenface [Turk & Pentland]

Image Understanding, Xuejin Chen Slide3

Eigenface

Projecting a new image into the subspace spanned by the

eigenfaces

(face space)Classifying the face by comparing its position in face space with the positions of known individuals Image Understanding, Xuejin Chen Slide4

Face image used as the training set

Image Understanding, Xuejin Chen Slide5

Average face

7

eigenfaces

calculated from the training imagesImage Understanding, Xuejin Chen Slide6

Calculating Eigenfaces

Face image : intensity matrix -D vector

PCA: to find the vectors that best account for the distribution of face images within the entire image space

Training imagesAverage faceEach face differs from the average by Seeks a set of M orthogonal vectors that best describe the distribution of the data

The

kth

vector is chosen such that

Image Understanding, Xuejin Chen Slide7

Calculating Eigenfaces

The vector and scalar are the eigenvectors and

eigenvalues

, respectively, of the covariance matrix Image Understanding, Xuejin Chen

Intractable task for typical image size

Need a

computationally feasible

method to find these eigenvectors Slide8

Calculating Eigenfaces

If image number M < N^2, only M meaningful eigenvectors

Consider eigenvectors vi of A’A such that

Image Understanding, Xuejin Chen

Solve

MxM

matrix

16x16

 16384*16384

(an 128x128 image)Slide9

Calculating Eigenfaces

Construct

MxM

matrixCompute M eigenvectors of LThe M eigenfaces is the linear combination of the M eigenvectors Image Understanding, Xuejin Chen

The computations are greatly reducedSlide10

Average face

7

eigenfaces

calculated from the training imagesImage Understanding, Xuejin Chen Slide11

Classify a Face Image

40

eigenfaces

are sufficient for a good description for an ensemble of M=115 images [Sirovich and Kirby 1987]A smaller M’ is sufficient for identification Choose

M’ significant eigenvectors of L matrix (M=16, M’=7)

Image Understanding, Xuejin Chen Slide12

Classify a Face Image

New image : transformed into its

eigenface

components Image Understanding, Xuejin Chen Slide13

Classify a Face Image

describe the contribution of each

eigenface

in representing the input image, treating the eigenfaces as a basis set for face imageFind a class of the faceA simple distance

Image Understanding, Xuejin Chen

Average of a class or individualSlide14

Classify a Face Image

Creating the weight vector is: projecting the original face image onto the low-D face space

Distance from image to face space

Image Understanding, Xuejin Chen Slide15

Four possibilities for an image and pattern vector

Near face space and near a face class

Near face space but not near a known face class

Distant from face space but near a face class

Distant from face space and not near a known face class

Image Understanding, Xuejin Chen Slide16

Distance to face space

29.8

58.5

5217.4

Image Understanding, Xuejin Chen Slide17

Summary of Eigenface

Recognition

Collect a set of characteristic face image of the known individuals.

The set should include a number of images for each person, with some variation in expression and lighting (M=40)Calculate the 40x40 matrix L

, find its

eigenvalues

and eigenvectors, choose M’(~10) eigenvectors with highest

eigenvalues

Compute

eigenfaces

For each known individual, calculate the class vector by averaging the

eigenface

pattern vector .

Choose a

threshold

that defines the maximum allowable distance from any face class and

a

threshold

that defines the maximum allowable distance from face space

Image Understanding, Xuejin Chen Slide18

Summary of Eigenface

Recognition

For each new face image to be identified, calculate

its pattern vector , the distance to face space , the distance to each known class . if the minimum distance Classify the input face as the individual associated with class vector

If the minimum distance

The image may be classified as unknown, and

Optionally used to begin a new face class

Optionally, if the new image is classified as a known individual, the image may be added to the original set of familiar face image, and the

eigenfaces

may be recalculated (steps 1-4)

Image Understanding, Xuejin Chen Slide19

Locating and Detecting Faces

Assume a centered face image, the same size as the training images and the

eigenfaces

Using face space to locate the face in imageImages of faces do not change radically when projected into the face space, while the projection of nonface images appears quite different> detect faces in a scene

Image Understanding, Xuejin Chen Slide20

Use face space to locate face

At every location in the image, calculate the distance between the local

subimage

and face space, which is used as a measure of ‘faceness’  a face map Image Understanding, Xuejin Chen

Expensive calculationSlide21

Face Map

A

subimage

at (x,y) Image Understanding, Xuejin Chen Slide22

Face Map

: linear combination of orthogonal

eigenface

vectors Image Understanding, Xuejin Chen Slide23

Face Map

Image Understanding, Xuejin Chen

Correlation operatorSlide24

Face Map

Image Understanding, Xuejin Chen

Precomputed

L+1 correlations

Can be implemented by a simple neural networksSlide25

Learning to Recognize New Faces

An image is close to face space, but not classified as one of the familiar faces, labeled as “unknown”

If a collection of “unknown” pattern vectors cluster in the pattern space, a new face is postulated

Check similarity: the distance from each image to the mean is smaller than a thresholdAdd the new face to database (optionally)

Image Understanding, Xuejin Chen Slide26

Background Issue

Eigenface

analysis can not distinguish the face from background

Segmentation? Multiply the image by a 2D Gaussian window centered on the face Deemphasize the outside of the face Also practical for hairstyle changingImage Understanding, Xuejin Chen Slide27

Scale and Orientation Issue

Recognition performance decreases quickly as the size is misjudged

Motion estimation?

Multiscale eigenfaces / multiscale input imageNon-upright facesOrientation estimation using symmetry operatorsImage Understanding, Xuejin Chen Slide28

Distribution in Face Space

Nearest-neighbor classification assumes Gaussian distribution of an individual feature vector

No prior to assume any distribution

Nonlinear networks to learn the distribution by example [Fleming and Cottrell, 1990] Image Understanding, Xuejin Chen Slide29

Multiple Views

Define a number of face classes for each person

Frontal view

Side view at ± 45° Right and left profile viewsImage Understanding, Xuejin Chen Slide30

Experiments

Database

Over 2500 face images under controlled conditions

16 subjectsAll combinations of 3 head orientations, 3 head sizes, 3 lighting conditions Construct 6-level Gaussian pyramid from 512x512 to 16x16 Image Understanding, Xuejin Chen Slide31

Image Understanding, Xuejin Chen

Variation of face images for one individualSlide32

Experiments with Lighting, Size, Orientation

Training sets

One image of each person, under the same lighting condition, size, orientation

Use seven eigenfacesMean accuracy as the difference between the training conditions, test conditionsDifference in illuminationImage size, Head orientation Combinations of illumination, size, orientation

Image Understanding, Xuejin Chen Slide33

Changing lighting conditions --- few errors

Image size changing -- performance dramatically drops

Need

multiscale approachImage Understanding, Xuejin Chen

(a) Lighting 96% (b) Size 85% (c) Orientation 64%

(d) Orientation & lighting (e)

Orienation

& Size 1 (f) Orientation & Size 2

(g) Size & Lighting 1 (h) Size & Lighting 2Slide34

Experiments with varying thresholds

Smaller threshold:

Few errors, but more false negative

Larger thresholdMore errorsTo achieve 100% accurate recognition, boost unknown rate to19% while varying lighting39% for orientation60% for size

Set the unknown rates to 20%, the correct recognition rate

100% for lighting

94% for orientation

74% for size

Image Understanding, Xuejin Chen Slide35

Neural Networks

Can be implemented using parallel computing elements

Image Understanding, Xuejin Chen Slide36

Collection of networks to implement computation of the pattern vector, projection into face space, distance from face space, and identification

Image Understanding, Xuejin Chen Slide37

Image Understanding, Xuejin Chen Slide38

Conclusion

Not general recognition algorithm

Practical and well fitted to face recognition

Fast and simple Do not require perfect identificationLow false-positive rateA small set of likely matches for user-interactionImage Understanding, Xuejin Chen Slide39

Eigenface

Tutorial

Image Understanding, Xuejin Chen Slide40

Bayesian Face Recognition

Baback

Moghaddam, Tony Jebaraand Alex Pentland

Pattern Recognition

33(11), Nov. 2000Slide41

Novelty

A direct visual matching of face images

Probabilistic measure of similarity

Bayesian (MAP) analysis of image differencesSimple computation of nonlinear Bayesian similarity Slide42

A Bayesian Approach

Many face recognition systems rely on similarity metrics

nearest-neighbor, cross-correlation

Template matchingWhich types of variation are critical in expressing similarity?Slide43

Probabilistic Similarity Measure

Intensity difference

Two classes of facial image variations

Intrapersonal variationsExtrapersonal variationsSimilarity measure

Can be estimated using likelihoods given by

Bayes

rule

Non-Euclidean similarity measureSlide44

A Bayesian Approach

First instance of non-Euclidean similarity measure for face recognition

A generalized extension of

Linear Discriminant Analysis FisherFaceHas computational and storage advantages over most linear methods for large databaseSlide45

Probabilistic Similarity Measures

Previous Bayesian analysis of facial appearance

3 different inter-image representations were analyzed using the binary formulation

XYI-warp modal deformation spectraXY-warp optical flow fieldsSimplified I-(intensity)-only image-based differenceSlide46

Probabilistic Similarity Measures

Intrapersonal variations

Images of the same individual with different expression, lighting conditions..

Extrapersonal variationsVariations when matching two individualsBoth are Gaussian-distributed, learn the likelihoods Slide47

Probabilistic Similarity Measures

Similarity score

The priors can be set as the portion of image number in the database or specified knowledge

Maximum Posterior (MAP) Slide48

Probabilistic Similarity Measures

M individuals: M classes

Many classification -> binary pattern classification

Maximum likelihood measureAlmost as effective as MAP in most casesSlide49

Subspace Density Estimation

Intensity difference vector: high dimensional

No sufficient independent training examples

Computational cost is very largeIntrinsic dimensionality or major degree-of-freedom of intensity difference is significantly smaller than NPCADivides the vector space R^N into two complementary subspaces [Moghaddam

&

Pentaland

]Slide50

Subspace Density Estimation

Two complementary subspaces

A typical

eigenvalue

spectrumSlide51

Subspace Density Estimation

Likelihood estimation

True marginal density in F

Estimated marginal density in F’Slide52

Dual

Eigenfaces

Intrapersonal

Extrapersonal

Variations mostly due to expression changes

Variations such as hair, facial hair, glasses…Slide53

Dual Eigenfaces

Intensity differences of

extrapersonal

type span a larger vector spaceIntrapersonal eigenfaces corresponds to a more tightly constrained subspaceThe key idea of probability similarity measureSlide54

Efficient Similarity Computation

Two classes: one intrapersonal and the other as

extrapersonal

variations with Gaussian distribution

Z

ero-mean since the pair

Use the principal componentsSlide55

CSE 576, Spring 2008

Face Recognition and Detection

55

Bayesian Face RecognitionSlide56

Computation

To get the similarity

Subtracting

Project to principal eigenfaces of both extrapersonal and intrapersonal GaussiansExpotentials for likelihoodIterated all the operations over all members of the database (many I-k images) until find the maximum scoreSlide57

Offline Transformations

Preprocess the

I_k

images with whitening transformations Consequently every image is stored as two vectors of whitened subspace coefficientsSlide58

Offline Transformations

Euclidean distances are computed times for each similarity

Likelihood

Avoid unnecessary and repeated image differencing and online projectionSlide59

Experiments

ARPA FERET database

Images taken at different times, location, imaging conditions (clothes, lighting)

Training Set74 pairs of images (2/person)Test set38 pairs of imagesSlide60

Differences in clothing, hair, lighting..

(a) Training Set

(b) Test Set Slide61

Face Alignment (Detection)Slide62

Comparison with EigenfacesSlide63

Bayesian Matching

Training data

74 intrapersonal differences

296 extrapersonal differencesSeparate PCA analysis on eachHow they distribute?Completely enmeshed distributions with the same principle components

Hard to distinguish low-amplitude

extrapersonal

difference from intrapersonal differenceSlide64

Separate PCA

Dealing with low-D hyper-ellipsoids with are intersecting near the origin of a very high-D space

Key distinguishing factor is their relative orientationSlide65

Dual

Eigenfaces

Intrapersonal

Extrapersonal

Variations mostly due to expression changes

Variations such as hair, facial hair, glasses…Slide66

Dual Eigenfaces

Intensity differences of

extrapersonal

type span a larger vector spaceIntrapersonal eigenfaces corresponds to a more tightly constrained subspaceThe key idea of probability similarity measureSlide67

Dual Eigenfaces

Compute two sets of , compute likelihood estimates

Use principal dimensions

Set the priors as equal Slide68

Performance

Improvement over the accuracy obtained with a standard

eigenface

nearest-neighbor matching ruleMaximum likelihood gets a similar result with MAP 2~3% deficit on recognition rateComputational cost is cut by a factor of 2 Slide69

Performance Slide70

Computation Simplification

Exact mapping of the probabilistic similarity score without requiring repeated image-differencing and

eigenface

projectionsNonlinar matching  simple Euclidean norms of their whitened feature vectors which can be

precomputed

offlineSlide71

Discussion

Model larger variations in facial appearance?

Pose, facial decorations?

Regular glasses Sunglasses, significant changes in beards, hairAdd more variation in interpersonal training? … Views View-based multiple model Slide72

CSE 576, Spring 2008

Face Recognition and Detection

72

Bayesian Face RecognitionSlide73

Conclusions

Good performance of probabilistic matching

Advantageous in intra/extra density estimates explicitly characterize the type of appearance variations

Discovering the principle modes of variationsOptimal non-linear decision rule Do not need to compute and store eigenfaces for each individual

One or two global

eigenfaces

are sufficient

Maximum Likelihood vs. MAPSlide74

View-Based and Modular Eigenspaces for Face Recognition

Alex Pentland, Baback Moghaddam and Thad Starner

CVPR’94Slide75

CSE 576, Spring 2008

Face Recognition and Detection

75

Part-based eigenfeaturesLearn a

separate

eigenspace

for

each face feature

Boosts

performance of regular

eigenfacesSlide76

CSE 576, Spring 2008

Face Recognition and Detection

76

Morphable Face ModelUse subspace to model elastic 2D or 3D

shape

variation (vertex positions), in addition to

appearance

variation

Shape S

Appearance TSlide77

CSE 576, Spring 2008

Face Recognition and Detection

77

Morphable Face Model

3D models from Blanz and Vetter ‘99Slide78

Project 3

Eigenfaces

Given a skeleton, you need to fill the functions

PCA to compute eigenfacesProjection into the face spaceDetermining if a vector represents a faceVerifying a user based on a face, finding a face match given a set of user face informationFinding the size and position of a face in an image

Image Understanding, Xuejin Chen Slide79

Project 3

Eigenfaces

Skeleton code is large, please take time to get familiar with the

classes and methodsVectorImage operationsMinimum modification: faces.cpp, eigenfaces.cppImage Understanding, Xuejin Chen