K Nearest Neighbor Classification - PowerPoint Presentation

1. K Nearest Neighbor Classification

2. Bayes Classifier: RecapLP( HILSA | L)P( TUNA | L)P( SHARK | L)Maximum Aposteriori (MAP) RuleDistributions assumed to be of particular family (e.g., Gaussian), and parameters estimated from training data.

3. Bayes Classifier: RecapL +- P( HILSA | L)P( TUNA | L)P( SHARK | L)Approximate Maximum Aposteriori (MAP) RuleNon-parametric (data driven) approach: consider a small window around L,Find which class is most populous in that window.

4. Nearest Neighbor ClassifiersBasic idea:If it walks like a duck, quacks like a duck, then it’s probably a duckTraining RecordsTest RecordCompute DistanceChoose k of the “nearest” records

5. Basic Ideak-NN classification rule is to assign to a test sample the majority category label of its k nearest training samplesIn practice, k is usually chosen to be odd, so as to avoid tiesThe k = 1 rule is generally called the nearest-neighbor classification rule

6. Definition of Nearest Neighbor K-nearest neighbors of a record x are data points that have the k smallest distance to x

7. Voronoi DiagramProperties:All possible points within a sample's Voronoi cell are the nearest neighboring points for that sampleFor any sample, the nearest sample is determined by the closest Voronoi cell edge

8. Distance-weighted k-NNReplace by: General Kernel functions like Parzen Windows may be considered Instead of inverse distance.

9. Predicting Continuous ValuesReplace by: Note: unweighted corresponds to wi=1 for all i

10. Nearest-Neighbor Classifiers: IssuesThe value of k, the number of nearest neighbors to retrieveChoice of Distance Metric to compute distance between recordsComputational complexitySize of training setDimension of data

11. Value of KChoosing the value of k:If k is too small, sensitive to noise pointsIf k is too large, neighborhood may include points from other classesRule of thumb:K = sqrt(N)N: number of training points

12. Distance Metrics

13. Distance Measure: Scale EffectsDifferent features may have different measurement scalesE.g., patient weight in kg (range [50,200]) vs. blood protein values in ng/dL (range [-3,3])ConsequencesPatient weight will have a much greater influence on the distance between samplesMay bias the performance of the classifier

14. StandardizationTransform raw feature values into z-scores is the value for the ith sample and jth feature is the average of all for feature j is the standard deviation of all over all input samplesRange and scale of z-scores should be similar (providing distributions of raw feature values are alike)

15. Nearest Neighbor : DimensionalityProblem with Euclidean measure:High dimensional data curse of dimensionalityCan produce counter-intuitive resultsShrinking density – sparsification effect1 1 1 1 1 1 1 1 1 1 1 00 1 1 1 1 1 1 1 1 1 1 11 0 0 0 0 0 0 0 0 0 0 00 0 0 0 0 0 0 0 0 0 0 1vsd = 1.4142d = 1.4142

16. Distance for Nominal Attributes

17. Distance for Heterogeneous DataWilson, D. R. and Martinez, T. R., Improved Heterogeneous Distance Functions, Journal of Artificial Intelligence Research, vol. 6, no. 1, pp. 1-34, 1997

18. Nearest Neighbour : Computational ComplexityExpensiveTo determine the nearest neighbour of a query point q, must compute the distance to all N training examplesPre-sort training examples into fast data structures (kd-trees)Compute only an approximate distance (LSH)Remove redundant data (condensing)Storage RequirementsMust store all training data PRemove redundant data (condensing)Pre-sorting often increases the storage requirementsHigh Dimensional Data“Curse of Dimensionality”Required amount of training data increases exponentially with dimensionComputational cost also increases dramaticallyPartitioning techniques degrade to linear search in high dimension

19. Reduction in Computational ComplexityReduce size of training setCondensation, editingUse geometric data structure for high dimensional search

20. Condensation: Decision RegionsEach cell contains one sample, and every location within the cell is closer to that sample than to any other sample.A Voronoi diagram divides the space into such cells. Every query point will be assigned the classification of the sample within that cell. The decision boundary separates the class regions based on the 1-NN decision rule.Knowledge of this boundary is sufficient to classify new points.The boundary itself is rarely computed; many algorithms seek to retain only those points necessary to generate an identical boundary.

21. CondensingAim is to reduce the number of training samplesRetain only the samples that are needed to define the decision boundaryDecision Boundary Consistent – a subset whose nearest neighbour decision boundary is identical to the boundary of the entire training setMinimum Consistent Set – the smallest subset of the training data that correctly classifies all of the original training dataOriginal dataCondensed dataMinimum Consistent Set

22. CondensingCondensed Nearest Neighbour (CNN)Initialize subset with a single (or K) training exampleClassify all remaining samples using the subset, and transfer any incorrectly classified samples to the subsetReturn to 2 until no transfers occurred or the subset is fullIncrementalOrder dependentNeither minimal nor decision boundary consistentO(n3) for brute-force method

23. CondensingCondensed Nearest Neighbour (CNN)Initialize subset with a single training exampleClassify all remaining samples using the subset, and transfer any incorrectly classified samples to the subsetReturn to 2 until no transfers occurred or the subset is full

24. CondensingCondensed Nearest Neighbour (CNN)Initialize subset with a single training exampleClassify all remaining samples using the subset, and transfer any incorrectly classified samples to the subsetReturn to 2 until no transfers occurred or the subset is full

25. CondensingCondensed Nearest Neighbour (CNN)Initialize subset with a single training exampleClassify all remaining samples using the subset, and transfer any incorrectly classified samples to the subsetReturn to 2 until no transfers occurred or the subset is full

26. CondensingCondensed Nearest Neighbour (CNN)Initialize subset with a single training exampleClassify all remaining samples using the subset, and transfer any incorrectly classified samples to the subsetReturn to 2 until no transfers occurred or the subset is full

27. CondensingCondensed Nearest Neighbour (CNN)Initialize subset with a single training exampleClassify all remaining samples using the subset, and transfer any incorrectly classified samples to the subsetReturn to 2 until no transfers occurred or the subset is full

28. CondensingCondensed Nearest Neighbour (CNN)Initialize subset with a single training exampleClassify all remaining samples using the subset, and transfer any incorrectly classified samples to the subsetReturn to 2 until no transfers occurred or the subset is full

29. High dimensional searchGiven a point set and a nearest neighbor query pointFind the points enclosed in a rectangle (range) around the queryPerform linear search for nearest neighbor only in the rectangleQuery

30. kd-tree: data structure for range searchIndex data into a treeSearch on the treeTree construction: At each level we use a different dimension to splitx=5 y=3y=6x=6ABCDEx<5x>=5

31. kd-tree exampleX=5y=5y=6x=3y=2x=8x=7X=5X=8X=7X=3Y=6Y=2

32. KNN: Alternate TerminologiesInstance Based LearningLazy LearningCase Based ReasoningExemplar Based Learning