Generalized Principal Component Analysis - PowerPoint Presentation

Slide1

Generalized Principal Component Analysis

René VidalCenter for Imaging ScienceInstitute for Computational MedicineJohns Hopkins UniversitySlide2

Data segmentation and clustering

Given a set of points, separate them into multiple groups

Discriminative methods: learn boundary

Generative methods: learn mixture model, using, e.g. Expectation MaximizationSlide3

Dimensionality reduction and clustering

In many problems data is high-dimensional: can reduce dimensionality using, e.g. Principal Component AnalysisImage compression

Recognition

Faces (Eigenfaces)

Image segmentation

Intensity (black-white)

Texture Slide4

Segmentation problems in dynamic vision

Segmentation of video and dynamic texturesSegmentation of rigid-body motionsSlide5

Segmentation problems in dynamic vision

Segmentation of rigid-body motions from dynamic texturesSlide6

Clustering data on non Euclidean spaces

Clustering data on non Euclidean spaces

Mixtures of linear spaces

Mixtures of algebraic varieties

Mixtures of Lie groups

“Chicken-and-egg” problems

Given segmentation, estimate models

Given models, segment the data

Initialization?

Need to combine

Algebra/geometry, dynamics and statisticsSlide7

Talk outline

Part I: TheoryPrincipal Component Analysis (PCA)Basic GPCA theory and algorithmsAdvanced statistical and algebraic methods for GPCAPart II: Applications

Applications to motion and video segmentation

Applications to dynamic texture segmentationSlide8

Principal Component Analysis (PCA)

Given a set of points x1, x2, …, xNGeometric PCA: find a subspace

S

passing through them

Statistical PCA: find projection directions that maximize the variance

Solution

(Beltrami’1873, Jordan’1874, Hotelling’33, Eckart-Householder-Young’36)

Applications: data compression, regression, computer vision (

eigenfaces), pattern recognition, genomics

Basis for

SSlide9

Extensions of PCA

Higher order SVD (Tucker’66, Davis’02)

Independent Component Analysis (Common ‘94)

Probabilistic PCA (Tipping-Bishop ’99)

Identify subspace from noisy data

Gaussian noise: standard PCA

Noise in exponential family (Collins et al.’01)

Nonlinear dimensionality reduction

Multidimensional scaling (Torgerson’58)

Locally linear embedding (Roweis-Saul ’00)

Isomap (Tenenbaum ’00)

Nonlinear PCA (Scholkopf-Smola-Muller ’98)

Identify nonlinear manifold by applying PCA to

data embedded in high-dimensional space

Principal Curves and Principal Geodesic Analysis (Hastie-Stuetzle’89, Tishbirany ‘92, Fletcher ‘04)Slide10

Generalized Principal Component Analysis

Given a set of points lying in multiple subspaces, identify

The number of subspaces and their dimensions

A basis for each subspace

The segmentation of the data points

“Chicken-and-egg” problem

Given segmentation, estimate subspaces

Given subspaces, segment the dataSlide11

Applications of GPCA in computer vision

GeometryVanishing pointsImage compression

Segmentation

Intensity (black-white)

Texture

Motion (2-D, 3-D)

Video (host-guest)

RecognitionFaces (Eigenfaces

)Man - WomanHuman GaitsDynamic Textures

Water-birdBiomedical imagingHybrid systems identificationSlide12

Prior work on subspace clustering

Iterative

algorithms

K-

subspaces

(

Kamhalta-Leen

‘94, Ho

et al. ’03

)

RANSAC, subspace selection and growing

(

Leonardis

et al. ’02

)

Probabilistic approaches

Mixtures of

PPCA

(Tipping-

Bishop ’99, Grubber-Weiss ’04)

Multi-Stage Learning

(

Kanatani

’

04)

Agglomerative

Lossy

Compression

(Ma et al. ’07)

Initialization

Geometric approaches: 2 planes in R

3

(

Shizawa

-Maze ’91)

Factorization-based approaches for

independent subspaces of equal

dimension

(

Boult

-Brown ‘91,

Costeira-Kanade

‘98,

Gear ’08,

Kanatani

’

01)

Spectral clustering based

approaches

(Zelnik-Manor ‘03, Yan

-

Pollefeys

’06, Fan’06, Chen-Lerman’08)Slide13

Basic ideas behind GPCA

Towards an analytic solution to subspace clusteringCan we estimate ALL models simultaneously using ALL data?When can we do so analytically? In closed form?Is there a formula for the number of models?Will consider the most general caseSubspaces of unknown and possibly different dimensions

Subspaces may intersect arbitrarily (not only at the origin)

GPCA is an algebraic geometric approach to data segmentation

Number of subspaces = degree of a polynomial

Subspace basis = derivatives of a polynomial

Subspace clustering is algebraically equivalent toPolynomial fitting

Polynomial differentiationSlide14

Introductory example: algebraic clustering in 1D

Number of groups?Slide15

Introductory example:

algebraic clustering in 1D

How to compute

n

,

c

,

b

’s?

Number of clusters

Cluster centers

Solution is unique if

Solution is closed form ifSlide16

Introductory example:

algebraic clustering in 2D

What about dimension 2?

What about higher dimensions?

Complex numbers in higher dimensions?

How to find roots of a polynomial of quaternions?

Instead

Project data onto one or two dimensional space

Apply same algorithm to projected data Slide17

Representing one subspace

One plane

One line

One subspace can be represented with

Set of linear equations

Set of polynomials of degree 1Slide18

De Morgan’s rule

Representing

n

subspaces

Two planes

One plane and one line

Plane:

Line:

A union of n subspaces can be represented with a set of homogeneous polynomials of degree nSlide19

Veronese map

Fitting polynomials to data points

Polynomials can be written linearly in terms of the vector of coefficients by using polynomial embedding

Coefficients of the polynomials can be computed from nullspace of embedded data

Solve using least squares

N = #data pointsSlide20

Finding a basis for each subspace

Case of hyperplanes: Only one polynomialNumber of subspacesBasis are normal vectors

Problems

Computing roots may be sensitive to noise

The estimated polynomial may not perfectly factor with noisy

Cannot be applied to subspaces of different dimensions

Polynomials are estimated up to change of basis, hence they may not factor, even with perfect data

Polynomial Factorization (GPCA-PFA) [CVPR 2003]

Find roots of polynomial of degree in one variable

Solve linear systems in variables

Solution obtained in closed form forSlide21

Finding a basis for each subspace

To learn a mixture of subspaces we just need one positive example per class

Polynomial Differentiation (GPCA-PDA) [CVPR’04]Slide22

Choosing one point per subspace

With noise and outliers Polynomials may not be a perfect union of subspacesNormals can estimated correctly by choosing points optimallyDistance to closest subspace without knowing segmentation?Slide23

GPCA for hyperplane segmentation

Coefficients of the polynomial can be computed from null space of embedded data matrixSolve using least squaresN = #data pointsNumber of subspaces can be computed from the rank of embedded data matrixNormal to the subspaces can be computed from the derivatives of the polynomialSlide24

GPCA for

subspaces of different dimensionsThere are multiple polynomials fitting the dataThe derivative of each polynomial gives a different normal vector

Can obtain a basis for the subspace by applying PCA to normal vectorsSlide25

GPCA

for

subspaces

of different dimensions

Apply polynomial embedding to projected data

Obtain multiple subspace model by polynomial fitting

Solve to obtain

Need to know number of subspaces

Obtain bases & dimensions by polynomial differentiation

Optimally choose one point per subspace using distanceSlide26

An example

Given data lying in the union of the two subspacesWe can write the union asTherefore, the union can be represented with the two polynomialsSlide27

An example

Can compute polynomials fromCan compute normals fromSlide28

Dealing with high-dimensional data

Minimum number of points

K = dimension of ambient space

n = number of subspaces

In practice the dimension of each subspace ki is much smaller than K

Number and dimension of the subspaces is preserved by a linear projection onto a subspace of dimension

Can remove outliers by robustly fitting the subspace

Open problem: how to choose projection?

PCA?

Subspace 1

Subspace 2Slide29

GPCA with spectral clustering

Spectral clustering

Build a similarity matrix between pairs of points

Use eigenvectors to cluster data

How to define a similarity for subspaces?

Want points in the same subspace to be close

Want points in different subspace to be far

Use GPCA to get basis

Distance: subspace anglesSlide30

Extensions of GPCA to nonlinear manifolds

Segmentation of quadratic forms (Rao-Yang-Ma ’05, ‘06)Segmentation of bilinear surfaces

(Vidal-Ma-

Soatto-Sastry

’03, ‘06)

Segmentation of mixed linear and bilinear

(

Singaraju-Vidal ‘06)Segmentation of

trilinear surfaces (Hartley-Vidal ’05, ’08)Slide31

Comparison of PFA, PDA, K-sub, EMSlide32

N

u

l

l

(

L

n

)

L

n

=

[

º

n

(

x

1

)

;

:

:

:

;

º

n

(

x

N

)

]

:

Dealing with outliers

GPCA with perfect data

GPCA with outliers

GPCA fails because PCA fails seek a robust estimate of whereSlide33

Three approaches to tackle outliers

Probability-based: small-probability samplesProbability plots: [Healy 1968, Cox 1968]PCs: [Rao 1964, Ganadesikan & Kettenring 1972]M-estimators: [Huber 1981, Camplbell 1980]Multivariate-trimming (MVT):

[Ganadesikan & Kettenring 1972]

Influence-based: large influence on model parameters

Parameter difference with and without a sample:

[Hampel et al. 1986, Critchley 1985]

Consensus-based: not consistent with models of high consensus.

Hough: [Ballard 1981, Lowe 1999]RANSAC: [Fischler & Bolles 1981, Torr 1997]

Least Median Estimate (LME): [Rousseeuw 1984, Steward 1999]Slide34

Robust GPCASlide35

Robust GPCA

Simulation on Robust GPCA (parameters fixed at = 0.3rad and = 0.4RGPCA – Influence

RGPCA - MVT

(e) 12%

(f) 32%

(g) 48%

(h) 12%

(i) 32%

(j) 48%

(k) 12%

(l) 32%

(m) 48%

(n) 12%

(o) 32%

(p) 48%Slide36

Robust GPCA

Comparison with RANSACAccuracy

Speed

(q) (2,2,1) in

3

(s) (5,5,5) in

6

(r) (4,2,2,1) in

5

Arrangement

(2,2,1) in

3

(4,2,2,1) in

5

(5,5,5) in

6

RANSAC

44s

5.1min

3.4min

MVT

46s

23min

8min

Influence

3min

58min

146min

Table

: Average time of RANSAC and RGPCA with 24% outliers. Slide37

Applications to motion and and video segmentation

René VidalCenter for Imaging ScienceInstitute for Computational Medicine

Johns Hopkins UniversitySlide38

3-D motion segmentation problem

Given a set of point correspondences in multiple views, determine

Number of motion models

Motion model: affine, homography, fundamental matrix, trifocal tensor

Segmentation: model to which each pixel belongs

Mathematics of the problem depends on

Number of frames (2, 3, multiple)

Projection model (affine, perspective)

Motion model (affine, translational, homography, fundamental matrix, etc.)

3-D structure (planar or not)Slide39

Single-body factorization

Affine camera model

p

= point

f

= frame

Motion of one rigid-body lives in a 4-D subspace

(

Boult and Brown ’91, Tomasi and

Kanade ‘92)P = #points

F = #frames

Structure

= 3D surface

Motion = camera position and orientationSlide40

Projection onto a 5-D subspace

Motion of one rigid-body lives in 4-D subspace of

By projecting onto a 5-D subspace of

Number and dimensions of subspaces are preserved

Motion segmentation is equivalent to clustering subspaces of dimension

2, 3 or 4 in

Minimum #frames = 3

(CK needs a minimum of 2n frames for n motions)

Can remove outliers by robustly fitting the 5-D subspace using Robust SVD (DeLaTorre-Black)

What projection to use?

PCA: 5 principal components

RPCA: with outliers

Motion 1

Motion 2Slide41

Hopkins 155 motion segmentation database

Collected 155 sequences120 with 2 motions35 with 3 motions

Types of sequences

Checkerboard sequences: mostly full

dimensional and independent motions

Traffic sequences: mostly degenerate (linear,

planar) and partially dependent motions

Articulated sequences: mostly full dimensional

and partially dependent motionsPoint correspondences

In few cases, provided by Kanatani & PollefeysIn most cases, extracted semi-automatically

with OpenCVSlide42

Experimental results: Hopkins 155 database

2 motions, 120 sequences, 266 points, 30 framesSlide43

Experimental results: Hopkins 155 database

3 motions, 35 sequences, 398 points, 29 framesSlide44

Spatial Temporal Video Segmentation

René VidalCenter for Imaging ScienceInstitute for Computational Medicine

Johns Hopkins UniversitySlide45

Modeling dynamic textures

Examples of dynamic textures: Model temporal evolution as the output of a linear dynamical system (LDS):

Soatto

et al. ‘01

dynamics

appearance

images

z

t

+

1

=

A

z

t

+

v

t

y

t

=

C

z

t

+

w

tSlide46

Segmentation of dynamic textures

Model intensity at each pixel as the output of an AR modelDefine regressors & parameters

Regressors

with same texture live in a

hyperplane

Multiple dynamic textures live in multiple

hyperplanes

Can cluster the

regressors

using GPCA

water

steam

A

hyperplaneSlide47

Segmenting non-moving dynamic textures

One dynamic texture lives in the observability subspace

Multiple textures live in multiple subspaces

Cluster the data using GPCA

water

steam

z

t

+

1

=

A

z

t

+

v

t

y

t

=

C

z

t

+

w

tSlide48

Segmentation of dynamic texturesSlide49

Variational segmentation of dynamic textures

Fixed boundary segmentation results and comparison

Ocean-smoke

Ocean-dynamics

Ocean-appearanceSlide50

Variational segmentation of dynamic textures

Moving boundary segmentation results and comparisonOcean-fireSlide51

Variational segmentation of dynamic textures

Results on a real sequence

Raccoon on RiverSlide52

Temporal video segmentationSlide53

Temporal video segmentation

Empty living roomMiddle-aged man enters

Woman enters

Young man enters, introduces the woman and leaves

Middle-aged man flirts with woman and steals her tiara

Middle-aged man checks the time, rises and leaves

Woman walks him to the door

Woman returns to her seatWoman misses her tiaraWoman searches her tiaraWoman sits and dismaysSlide54

Conclusions

Many problems in computer vision can be posed as subspace clustering problemsTemporal video segmentation2-D and 3-D motion segmentation

Dynamic texture segmentation

Nonrigid motion segmentation

These problems can be solved using GPCA: an algorithm for clustering subspaces

Deals with unknown and possibly different dimensions

Deals with arbitrary intersections among the subspaces

GPCA is based on

Projecting data onto a low-dimensional subspace

Recursively fitting polynomials to projected subspacesDifferentiating polynomials to obtain a basisSlide55

References: Springer-Verlag 2008Slide56

Slides, MATLAB code, papers

http://perception.csl.uiuc.edu/gpcaSlide57

For more information,

Vision Lab @ Johns Hopkins UniversityThank

You!

NSF CNS-0809101

NSF ISS-0447739

ARL Robotics-CTA

ONR N00014-05-10836

NSF CNS-0509101

JHU APL-934652

NIH RO1 HL082729

WSE-APL

NIH-NHLBI