and MDS Wilson A Florero Salinas Dan Li Math 285 Fall 2015 1 Outline What is an outofsample extension O utofsample extension of PCA KPCA MDS 2 What is outofsampleextension ID: 551894
Download Presentation The PPT/PDF document "Out of sample extension of PCA, Kernel P..." is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
Out of sample extension of PCA, Kernel PCA, and MDS
Wilson A. Florero-SalinasDan LiMath 285, Fall 2015
1Slide2
Outline
What is an out-of-sample extension?Out-of-sample extension ofPCAKPCA
MDS
2Slide3
What is out-of-sample-extension?
Suppose we perform a dimensionality reduction on a data set New data becomes available.
Two options:
Option 1
: Re-train the model including the new available data.
Option 2
: Embed the new data into the existing space obtained by the training data set.
3
out-of-sample extension
Q
uestion:
Why is this important?
Slide4
Principal Component Analysis (PCA)
4Slide5
Principal Component Analysis (PCA)
5Slide6
Out-of-sample PCA
Suppose new data becomes available.6Slide7
7
Out-of-sample PCASlide8
Kernel PCA
8
Main Idea:
Apply PCA in
the feature
space
Apply PCA
Solve eigenvalue problem
Center here, tooSlide9
Data is linearly separated when projected to a higher dimensional space.
9
Main Idea:
Apply PCA in a higher dimensional space
Kernel PCASlide10
Once in the higher dimension, proceed
like in the PCA case10Out of sample Kernel PCA
Center the data
New data
:Slide11
Out of sample Kernel PCA
Project new data into feature space which is obtained by the training data set.Apply the kernel trick
11Slide12
Out of sample Kernel PCA Demo
12
Green points are new dataSlide13
Multidimensional Scaling (MDS)
MDS visualizes a set of high dimensional data in lower dimensions based on their pairwise distances. The idea is to make pairwise distance of the data in the low dimension close to the original pairwise
data
In other words, two points
that are far
apart in
higher dimension stay far apart in the reduced dimension. Similarly, points that are close in distance will be mapped together in the reduced dimension.
13Slide14
Comparison of PCA and MDS
The purpose of the two methods is to find the most accurate data representation in a lower dimensional space. MDS preserves the most ‘similarities’ of the original data set.
In compare, PCA preserves most of the variance of the data.
14Slide15
Multidimensional Scaling (MDS)
Main idea: The new coordinates of the projected data can be derived by eigenvalue decomposition of the centered D matrix.
15Slide16
Similar to Kernel PCA, we can project the new data as follows：
16
d = [d
1,
d
2,
…
d
n
]
New point comes in
D=[d
ij
2
]
Out of sample MDSSlide17
Out of sample MDS
Similar to Kernel PCA, we can project the new data as follows
17
Main idea:Slide18
18
Out of sample MDS Demo 1Chinese cities dataSlide19
19
Out of sample MDS Demo 2Chinese cities dataSlide20
20
Question?Slide21
21Slide22
22
Out of sample MDS Demo 2
Seeds
data