DP-space: Bayesian Nonparametric Subspace Clustering with S - PowerPoint Presentation

Slide1

DP-space: Bayesian Nonparametric Subspace Clustering with Small-variance Asymptotics

Yining Wang, Jun zhuCarnegie Mellon UniversityTsinghua University

1Slide2

Subspace Clustering

2Slide3

Subspace Clustering Applications

Motion Trajectories tracking1

1 (

Elhamifar and Vidal, 2013), (Tomasi and

Kanade

, 1992)

3Slide4

Subspace Clustering Applications

Face Clustering11

(Elhamifar and Vidal, 2013), (Basri

and Jacobs, 2003)

Network hop counts, movie ratings, social graphs, …

4Slide5

Mixtures of Probabilistic PCA (MPPCA)

(Tipping and Bishop, 1999)Generative model:Observations:

Parameters: low-dimensional subspaces

and offsets

Likelihood model:

Mixture probabilities:

5Slide6

Mixtures of Probabilistic PCA (MPPCA)

Limitations: both number of clusters () and dimensions of subspaces (

) need to be pre-specified

Cross-validation?Challenge: exponential

number of configuration

Bayesian nonparametric extension:

Chinese Restaurant Process (CRP) over cluster assignments

Exponential prior over subspace rank

6Slide7

Dirichlet Process of PCA (DP-PCA)

7Slide8

Partially collapsed Gibbs sampling

Update of offset

Update of cluster assignment

Update of subspace dimension

(collapsed step)

Non-conjugate

8Slide9

Small-variance asymptotics (SVA)

Obtain simple, deterministic update rules under limiting behavior of algorithm hyper-parameters (e.g., variance)A toy example: limiting distribution of spherical GMM

The generative model:

Exact and limiting posterior on

:

Applications: significant increase in recent years

DPM (

Kulis

and Jordan, 2012), IBP (Broderick,

Kulis

and Jordan, 2013), Infinite HMM (

Roychowdhury

and Jordan, 2013), Infinite SVM (Wang and Zhu, 2014), ……

9

GMM

K-meansSlide10

SVA for DP-space

Update of cluster assignment :Before SVA

SVA:

,

After SVA

10

DP-means:

 Slide11

SVA for DP-space

Update of subspace rank

Before SVA:

SVA:

,

,

After SVA:

11Slide12

DP-space: deterministic update rules

Step 1 Update offsets:

Step 2

Update low-rank subspaces

Step 3

Update cluster assignments

12

K-plane

algo

.

Nonparametric

extensionSlide13

DP-space: a MAD-Bayes perspective

DP-space iteratively minimizes the following deterministic loss:

Data fitting term

Regularization

13Slide14

Experiments: a toy example

14Slide15

Experiments: a toy example

15Slide16

Experiments: Hopkins-155

16Slide17

Conclusion

A Bayesian nonparametric subspace clustering model that allows flexible cluster number and dimensionDerivation of a partially collapsed Gibbs samplerAn efficient deterministic iterative algorithm based on small-variance asymptotic analysis

Experimental results on real-world datasets: comparable performance with state-of-the-art methods while running much faster

17Slide18

References

R. Basri and D. Jacobs. Lambertian Reflection and Linear Subspaces. IEEE TPAMI, 2003.T. Broderick, B. Kulis

and M. Jordan. MAD-Bayes: MAP-based Asymptotic Derivations from Bayes. ICML, 2013.E. Elhamifar

and R. Vidal. Sparse Subspace Clustering: Algorithm, Theory and Applications. IEEE TPAMI, 2013B. Kulis

and M. Jordan. Revisiting k-means: New Algorithms via Bayesian

Nonparametrics

.

ICML,

2012.

A.

Roychowdhury

, K. Jiang and B.

Kulis

. Small-variance

Asymptotics

for Hidden Markov Models.

NIPS

, 2013.

M. Tipping and C. Bishop. Mixtures of Probabilistic Principle Component Analyzers

.

Neural Computation, 1999.C. Tomasi and T. Kanade. Shape and Motion from Image Streams under Orthography. IJCV, 1992.Y. Wang and J. Zhu. Small-variance Asymptotics for Dirichlet Process Mixtures of SVMs. AAAI, 2014.18