Noisy Sparse Subspace Clustering with dimension reduction - PowerPoint Presentation

Slide1

Noisy Sparse Subspace Clustering with dimension reduction

Yining Wang, Yu-Xiang Wang, Aarti SinghMachine Learning DepartmentCarnegie mellon university

1Slide2

Subspace Clustering

2Slide3

Subspace Clustering Applications

Motion Trajectories tracking1

1 (

Elhamifar and Vidal, 2013), (Tomasi and

Kanade, 1992)

3Slide4

Subspace Clustering Applications

Face Clustering11 (

Elhamifar and Vidal, 2013), (

Basri and Jacobs, 2003)

Network hop counts, movie ratings, social graphs, …

4Slide5

Sparse Subspace Clustering

(Elhamifar and Vidal, 2013), (Wang and Xu, 2013).Data:

Key idea:

similarity graph

based on

l

1

self-regression

No False Connections

5Slide6

Sparse Subspace Clustering

(Elhamifar and Vidal, 2013), (Wang and Xu, 2013).Data:

Key idea:

similarity graph

based on

l

1

self-regression

s.t.

Noiseless

data

Noisy

data

6Slide7

SSC with dimension reduction

Real-world data are usually high-dimensionalHopkins-155:

Extended Yale Face-B:

Computational concerns

Data availability: values of some features might be missingPrivacy concerns: releasing the raw data might cause privacy breaches.

7Slide8

SSC with dimension reduction

Dimensionality reduction:

How small can

p

be?

A

trivial result:

is OK.

L

: the number of subspaces (clusters)

r

: the intrinsic dimension of each subspace

Can we do better?

8Slide9

Main Result

,if

is a

subspace embedding

Random Gaussian projection

Fast Johnson-

Lindenstrauss

Transform (FJLT)

Uniform row subsampling under incoherence conditions

Sketching

……

Lasso SSC should be used

even if data are noiseless.

9Slide10

Proof sketch

Review of deterministic success conditions for SSC (Soltanolkotabi and Candes, 12)(Elhamifar

and Vidal, 13)Subspace incoherence

InradiusAnalyze perturbation under dimension reduction

Main results for noiseless and noisy cases.10Slide11

Review of SSC success condition

Subspace incoherenceCharacterizing inter-subspace separation

where

solves

Dual problem of Lasso SSC

11

Lasso SSC formulationSlide12

Review of SSC success condition

InradiusCharacterzing inner-subspace data point distribution

Large

inradius

Small

inradius

12Slide13

Review of SSC success condition

(Soltanolkotabi &

Candes, 2012)Noiseless SSC succeeds (similarity graph has no false connection) if

With dimensionality reduction:

Bound

13Slide14

Perturbation of subspace incoherence

We know that

…

So

because of

strong convexity

14Slide15

Perturbation of

inradius  

Main idea: linear operator transforms a ball to an ellipsoid

15Slide16

Main result

SSC with dimensionality reduction succeeds (similarity graph has no false connection) if

Error of approximate isometry

O(1) if

16

Regularization parameter of Lasso

Lasso SSC required even for noiseless problem

Takeaways: the

geometric gap

is a resource that can be traded-off for data dimension reduction

Noisy case: (

is the adversarial noise level)

 Slide17

Simulation results (Hopkins 155)

17Slide18

Conclusion

SSC provably succeeds with dimensionality reductionDimension after reduction can be as small as

Lasso SSC is required for provable results.

Questions?

18Slide19

References

M. Soltanolkotabi and E. Candes. A Geometric Analysis of Subspace Clustering with Outliers. Annals of Statistics, 2012.E.

Elhamifar and R. Vidal. Sparse Subspace Clustering: Algorithm, Theory and Applications. IEEE TPAMI, 2013

C. Tomasi and T. Kanade. Shape and Motion from Image Streams under Orthography.

IJCV, 1992.R. Basri and D. Jacobs. Lambertian

Reflection and Linear Subspaces.

IEEE TPAMI

, 2003.

Y.-X., Wang and H., Xu. Noisy Sparse Subspace Clustering.

ICML

, 2013.

19