Dimensionality and - PDF document

Dimensionality and dimensionalityreduction dimensionality Nuno Vasconcelos ECE De p artment , UCSD p, Note this course requires a it is responsibility to define it (although we can talk) If you are too far from this, nextlandmarkisa projectproposal next a two page, describing the main idea y y not cast in stone, but bad idea to delay 2 High dimensional spaces are strange! firstthingtoknow: first “never trust your intuition in high dimensions!” more often than not you will be wrong! th l fthi th ere are many examp f s we will do a couple 4 Hyper-cube vs hyper-sphere next consider the hyper-cube [-a,a] yp e r i.e. yp p , a a-a what does your intuition tell you about the relative j ects? volume of sphere ~= volume of cubevolume of ��sphere volume of cube volume of sphere volume of cube Hyper-cube vs hyper-sphere as the dimension increases the volume of the shaded corners becomes lar g e r g a a-a in high dimensions the picture you should have in mind is all the volume of the cube is in these spikes! 9 But there is more consider the crust of wecancomputeits its 1 S 2 no matter how small is, ratio goes to zero as d increases ie “ allthevolumeisinthecrust! ” e High dimensional Gaussian Homework: show that if and one considers the hyper-sphere where the p robabilit densit y dro p s to 1% of p eak value pyypp probability mass outside this sphere is where 2 (n) isachi - squaredrandomvariablewith n 12 a - n degrees of freedom The curse of dimensionality typical observation in Bayes decision theory:•error increases when number of features is large highly unintuitive since, theoretically:if I have a problem in n-D I can always generate a problem in (n+1) Dwithsmallerprobabilityoferror (n+1) - D e.g. two uniform classes in 1D A B A B can be transformed into a 2D problem with same error justaddanon - informativevariabley. a y. Curse of dimensionality in fact, it is impossible to do worse in 2D than 1D x if we move the classes along the lines shown in green the 16 Dimensionality reduction what do we do about this? we unnecessary can be measured in two ways:features are not discriminat features are not independent non-discriminant means that they do not separate the lll c l asses we ll discriminant non-discriminant Dimensionality reduction dependent features, even if very discriminant, are not needed -one is enou g h! e.g. data-mining company studying consumer credit card X = {salary, mortgage, car loan, # of kids, profession, ...} the first three features tend to be highly correlated: “the more you make, the higher the mortgage, the more expensive the car you drive”from one of these variables I can p redict the others ver y well py including features 2 and 3 does not increase the discrimination, but increases the dimension and leads to dittit 19 y es ti ma Principal component analysis if the data lives in a subspace, it is going to look very flat when viewed from the full space, e.g. 1D subspace in 2D 2D subspace in 3D this means that if we fit a Gaussian to the data the equiprobability contours are going to be highly skewed ellipsoids 21 Gaussian review introduce the transformation y = is orthonormal this is just a rotationand we have z 2 y 2 2 z 2 y = 1 y 2 12 1z1 2y1 obtain a rotated ellipse with principal components note thatis the Principal component analysis If y is Gaussian with covariance y2 •principal components iare the eigenvectors of • p al len g ths are the 12 y 12 ppg eigenvalues of bycomputingtheeigenvaluesweknowifthedataisflat : not flat y2 y2 12 y 1 2 y 1 Principal component analysis 27 Principal components face examples 29 31