0K - views

Exact Nearest Neighbor Algorithms

Published bykittie-lecroy

Exact Nearest Neighbor Algorithms Sabermetrics One of the best players ever .310 batting average 3,465 hits 260 home runs 1,311 RBIs 14x All-star 5x World Series winner Who is the next Derek Jeter? Derek Jeter

Tags : trees nearest points neighbor nearest trees neighbor points space distance wikipedia source image centers data point closest algorithm set
Embed :
Presentation Download Link

Download Presentation - The PPT/PDF document "Exact Nearest Neighbor Algorithms" is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.

Exact Nearest Neighbor Algorithms






Presentation on theme: "Exact Nearest Neighbor Algorithms"— Presentation transcript:

Exact Nearest Neighbor Algorithms

Sabermetrics One of the best players ever .310 batting average 3,465 hits260 home runs1,311 RBIs14x All-star5x World Series winnerWho is the next Derek Jeter? Derek Jeter Source: Wikipedia

Sabermetrics Classic example of nearest neighbor application Hits, Home runs, RBIs, etc. are dimensions in “Baseball-space” Every individual player has a unique point in this spaceProblem reduces to finding closest point in this space

POI Suggestions Simpler example, only 2d space Known set of interest points in a map - we want to suggest the closest Note: we could make this more complicated; we could add dimensions for ratings, category, newness, etc.How do we figure out what to suggest?Brute force: just compute distance to all known points and pick lowestO(n) in the number of points Feels wasteful… why look at the distance to the Eiffel Tower when we know you’re in NYC?Space partitioning

2 d trees 1 2 3 4 5 6 7 8 1 2 3 4 5 x x x x x x x (4,3)

2 d trees 1 2 3 4 5 6 7 8 1 2 3 4 5 x x x x x x x (4,3) (6,2) (1,4)

2 d trees 1 2 3 4 5 6 7 8 1 2 3 4 5 x x x x x x x (4,3) (1,4) (6,2) (2,2) (3,5)

2 d trees 1 2 3 4 5 6 7 8 1 2 3 4 5 x x x x x x x (4,3) (1,4) (6,2) (2,2) (3,5) (7,2) (8,5)

2 d trees 1 2 3 4 5 6 7 8 1 2 3 4 5 x x x x x x x (4,3) (1,4) (6,2) (2,2) (3,5) (7,2) (8,5)

2d trees - Nearest Neighbor 1 2 3 4 5 6 7 8 1 2 3 4 5 x x x x x x x (4,3) (1,4) (6,2) (2,2) (3,5) (7,2) (8,5) x

2d trees - Nearest Neighbor 1 2 3 4 5 6 7 8 1 2 3 4 5 x x x x x x x (4,3) (1,4) (6,2) (2,2) (3,5) (7,2) (8,5) x

2d trees - Nearest Neighbor 1 2 3 4 5 6 7 8 1 2 3 4 5 x x x x x x x (4,3) (1,4) (6,2) (2,2) (3,5) (7,2) (8,5) x

2d trees - Nearest Neighbor 1 2 3 4 5 6 7 8 1 2 3 4 5 x x x x x x x (4,3) (1,4) (6,2) (2,2) (3,5) (7,2) (8,5) √10 x

2d trees - Nearest Neighbor 1 2 3 4 5 6 7 8 1 2 3 4 5 x x x x x x x (4,3) (1,4) (6,2) (2,2) (3,5) (7,2) (8,5) √10 x

2d trees - Nearest Neighbor 1 2 3 4 5 6 7 8 1 2 3 4 5 x x x x x x x (4,3) (1,4) (6,2) (2,2) (3,5) (7,2) (8,5) √10 √5 x

2d trees - Nearest Neighbor 1 2 3 4 5 6 7 8 1 2 3 4 5 x x x x x x x (4,3) (1,4) (6,2) (2,2) (3,5) (7,2) (8,5) x √5 3

2d trees - Nearest Neighbor 1 2 3 4 5 6 7 8 1 2 3 4 5 x x x x x x x (4,3) (1,4) (6,2) (2,2) (3,5) (7,2) (8,5) x √5 3

2d trees - Nearest Neighbor 1 2 3 4 5 6 7 8 1 2 3 4 5 x x x x x x x (4,3) (1,4) (6,2) (2,2) (3,5) (7,2) (8,5) x √5

2d trees - Nearest Neighbor 1 2 3 4 5 6 7 8 1 2 3 4 5 x x x x x x x (4,3) (1,4) (6,2) (2,2) (3,5) (7,2) (8,5) x √5 √2

2d trees - Nearest Neighbor 1 2 3 4 5 6 7 8 1 2 3 4 5 x x x x x x x (4,3) (1,4) (6,2) (2,2) (3,5) (7,2) (8,5) x √2 2

2d trees - Nearest Neighbor 1 2 3 4 5 6 7 8 1 2 3 4 5 x x x x x x x (4,3) (1,4) (6,2) (2,2) (3,5) (7,2) (8,5) x √2 2

2d trees - Nearest Neighbor 1 2 3 4 5 6 7 8 1 2 3 4 5 x x x x x x x (4,3) (1,4) (6,2) (2,2) (3,5) (7,2) (8,5) x √2

2d trees - Nearest Neighbor 1 2 3 4 5 6 7 8 1 2 3 4 5 x x x x x x x (4,3) (1,4) (6,2) (2,2) (3,5) (7,2) (8,5) x

2d trees - Nearest Neighbor 1 2 3 4 5 6 7 8 1 2 3 4 5 x x x x x x x (4,3) (1,4) (6,2) (2,2) (3,5) (7,2) (8,5) x

2d trees - Nearest Neighbor 1 2 3 4 5 6 7 8 1 2 3 4 5 x x x x x x x (4,3) (1,4) (6,2) (2,2) (3,5) (7,2) (8,5) x

2d trees - Nearest Neighbor 1 2 3 4 5 6 7 8 1 2 3 4 5 x x x x x x x (4,3) (1,4) (6,2) (2,2) (3,5) (7,2) (8,5) x √5

2d trees - Nearest Neighbor 1 2 3 4 5 6 7 8 1 2 3 4 5 x x x x x x x (4,3) (1,4) (6,2) (2,2) (3,5) (7,2) (8,5) x √5

2d trees - Nearest Neighbor 1 2 3 4 5 6 7 8 1 2 3 4 5 x x x x x x x (4,3) (1,4) (6,2) (2,2) (3,5) (7,2) (8,5) x √5 √10

2d trees - Nearest Neighbor 1 2 3 4 5 6 7 8 1 2 3 4 5 x x x x x x x (4,3) (1,4) (6,2) (2,2) (3,5) (7,2) (8,5) x √5 √10 3

2d trees - Nearest Neighbor 1 2 3 4 5 6 7 8 1 2 3 4 5 x x x x x x x (4,3) (1,4) (6,2) (2,2) (3,5) (7,2) (8,5) x √5 √10 3

2d trees - Nearest Neighbor 1 2 3 4 5 6 7 8 1 2 3 4 5 x x x x x x x (4,3) (1,4) (6,2) (2,2) (3,5) (7,2) (8,5) x √5 √10

2d trees - Nearest Neighbor 1 2 3 4 5 6 7 8 1 2 3 4 5 x x x x x x x (4,3) (1,4) (6,2) (2,2) (3,5) (7,2) (8,5) x √5 √10 3

2d trees - Nearest Neighbor 1 2 3 4 5 6 7 8 1 2 3 4 5 x x x x x x x (4,3) (1,4) (6,2) (2,2) (3,5) (7,2) (8,5) x √5 √10 3

2d trees - Nearest Neighbor 1 2 3 4 5 6 7 8 1 2 3 4 5 x x x x x x x (4,3) (1,4) (6,2) (2,2) (3,5) (7,2) (8,5) x √5 √10 3 √17

2d trees - Nearest Neighbor 1 2 3 4 5 6 7 8 1 2 3 4 5 x x x x x x x (4,3) (1,4) (6,2) (2,2) (3,5) (7,2) (8,5) x √5 √10 3 √17

2d trees - Nearest Neighbor 1 2 3 4 5 6 7 8 1 2 3 4 5 x x x x x x x (4,3) (1,4) (6,2) (2,2) (3,5) (7,2) (8,5) x √5 √10 3 √17 2

2d trees - Nearest Neighbor 1 2 3 4 5 6 7 8 1 2 3 4 5 x x x x x x x (4,3) (1,4) (6,2) (2,2) (3,5) (7,2) (8,5) x √5 √10 3 √17 2

2d trees - Nearest Neighbor 1 2 3 4 5 6 7 8 1 2 3 4 5 x x x x x x x (4,3) (1,4) (6,2) (2,2) (3,5) (7,2) (8,5) x √5 √10 3 √17 2 √18

2d trees - Nearest Neighbor 1 2 3 4 5 6 7 8 1 2 3 4 5 x x x x x x x (4,3) (1,4) (6,2) (2,2) (3,5) (7,2) (8,5) x √5 √10 3 √17 2 √18

2d trees Construction complexity: O(n log n) Complication: how do you decide how to partition?Requires you be smart about picking pivotsAdding/Removing element: O(log n)This is because we know it’s balanced from the median selection...except adding/removing might make it unbalanced -- there are variants that handle this Nearest NeighborAverage case: O(log n) Worst case: O(n)Not great… but not worse than brute forceCan also be used effectively for range finding

k -d trees 2d trees can be extended to k dimensions Image Source: Wikipedia

k -d trees Same algorithm for nearest neighbor! Remember sabermetrics!...except there’s a catchCurse of DimensionalityThe higher the dimensions, the “sparser” the data gets in the spaceHarder to rule out portions of the tree, so many searches end up being fancy brute forces In general, k-d trees are useful when N >> 2k

Sabermetrics (reprise) Finding single neighbor could be noise-prone Are we sure this year’s “Derek Jeter” will be next year’s too? What if there are lots of close points… are we sure that the relative distance matters?Could ask for set of most likely playersAlternate question: will a player make it to the Hall of Fame?Still k-dimensional space, but we’re not comparing with individual point Am I in the “neighborhood” of Hall of Famers?Classic example of “classification problem”

k -Nearest Neighbors (kNN) New plan: find the k closest pointsEach can “vote” for a classification…or you can do some other kind of averagingCan we modify our k-d tree NN algorithm to do this?Track k closest points in max-heap (priority queue)Keep heap at size kOnly need to consider k’th closest point for tree pruning

Voronoi Diagrams Useful visualization of nearest neighbors Good when you have a known set of comparison pointsWide ranging applicationsEpidemiologyCholera victims all near one water pumpAviation Nearest airport for flight diversionNetworkingCapacity derivation RoboticsPoints are obstacles, edges are safestpaths Image Source: Wikipedia

Voronoi Diagrams Also helpful for visualizing effects of different distance metrics Image Source: Wikipedia Euclidean distance Manhattan distance

Voronoi Diagrams Polygon construction algorithm is a little tricky, but conceptually you can think of expanding balls around the points Image Source: Wikipedia

k -means Clustering Goal: Group n data points into k groups based on nearest neighborAlgorithm:Pick k data points at random to be starting “centers,” call each center ciFor each node n, calculate which of the k centers is the nearest neighbor and add it to set SiCompute the mean of all points in Si to generate a new ciGo back to (2) and repeat with the new centers, until the centers converge

k -means clustering Image Source: Wikipedia Notice: the algorithm basically creates Voronoi diagrams for the centers!

k -means clustering Does this always converge? Depends on distance function. Generally yes for EuclideanConverges quickly in practice, but worst case can take an exponential number of iterationsDoes it give the optimal clustering?NO! Well, at least not always. Image Source: Wikipedia

Other space partitioning data structures Leaf point k -d treesOnly stores points in leaves, but leaves can store more than one pointSplit space at the middle of longest axisEffectively “buckets” points - can be used for approximate nearest neighborQuadtreesSplit space into quadrants (i.e. every tree node has four children)Quadrant can only contain at most q nodesIf there are more than q, split that quadrant again into quadrantsApplications Collision detection (video games)Image representation/processing (transforming/comparing/etc. nearby pixels)Sparse data storage (spreadsheets) Octrees are extension to 3d

Exact Nearest Neighbor Algorithms - Description

Exact Nearest Neighbor Algorithms Sabermetrics One of the best players ever 310 batting average 3465 hits 260 home runs 1311 RBIs 14x Allstar 5x World Series winner Who is the next Derek Jeter Derek Jeter ID: 765222 Download Presentation

Similar presentations

Approximate nearest neighbor for
Approximate nearest neighbor for

ℓ. p. –spaces (2<p<∞) via . embeddings. Yair. . Bartal. . Lee-Ad Gottlieb Hebrew U. Ariel University. Nearest neighbor search. Problem definition:. Given a set of points S, preprocess S so that the following query can be answered efficiently:.

Fast Nearest
Fast Nearest

Neighbor. Search with Keywords. Abstract. Conventional spatial queries, such as range search and nearest . neighbor. retrieval, involve only conditions on objects' geometric properties. Today, many modern applications call for novel forms of queries that aim to find objects satisfying both a spatial predicate, and a predicate on their associated texts. For example, instead of considering all the restaurants, a nearest .

Processing Transitive Nearest-Neighbor Queries in Multi-Cha
Processing Transitive Nearest-Neighbor Queries in Multi-Cha

Xiao Zhang. 1. , Wang-Chien Lee. 1. , Prasenjit Mitra. 1, 2. , Baihua Zheng. 3. 1. Department of Computer Science and Engineering. 2. College of Information Science and Technology. The Pennsylvania State University.

JMLR Workshop and Conference Proceedings vol Randomized partition trees for exact nearest neighbor search Sanjoy Dasgupta dasguptacs
JMLR Workshop and Conference Proceedings vol Randomized partition trees for exact nearest neighbor search Sanjoy Dasgupta dasguptacs

ucsdedu Department of Computer Science and Engineering University of California San Diego 9500 Gilman Drive La Jolla CA 92093 Kaushik Sinha kaushiksinhawichitaedu Department of Electrical Engineering and Computer Science Wichita State University 1845

Efficient Evaluation of k-Range Nearest Neighbor Queries in
Efficient Evaluation of k-Range Nearest Neighbor Queries in

Jie Bao Chi-Yin Chow Mohamed F. Mokbel. Department of Computer Science and Engineering. University of Minnesota – Twin Cities. Wei-Shinn Ku. Department of Computer Science and Software Engineering.

Image Recognition
Image Recognition

Christian Cosgrove. Kelly. Li. Rebecca. Lin. Shree . Nadkarni. Samanvit. . Vijapur. Priscilla. Wong. Yanjun. Yang. Kate Yuan. Daniel Zheng. Drew . University. New . Jersey Governor’s School in the Sciences.

NearOptimal Hashing Algorithms for Approximate Nearest Neighbor in High Dimensions by Alexandr Andoni and Piotr Indyk The goal of this article is twofold
NearOptimal Hashing Algorithms for Approximate Nearest Neighbor in High Dimensions by Alexandr Andoni and Piotr Indyk The goal of this article is twofold

In the first part we survey a family of nearest neighbor algorithms that are based on the concept of locality sensitive hashing Many of these algorithm have already been successfully applied in a variety of practical scenarios In the second part of

DATA MINING
DATA MINING

LECTURE 10. Classification. . k-nearest neighbor classifier. . Naïve Bayes. . Logistic Regression. . Support Vector Machines. NEAREST NEIGHBOR CLASSIFICATION. Instance-Based Classifiers. Store the training records .

Scalable Nearest Neighbor Algorithms for High Dimensional Data Marius Muja Member IEEE and David G
Scalable Nearest Neighbor Algorithms for High Dimensional Data Marius Muja Member IEEE and David G

Lowe Member IEEE Abstract For many computer vision and machine learning problems large training sets are key for good performance However the most computationally expensive part of many computer vision and machine learning algorithms consists of 642

Nearest Neighbor Editing and
Nearest Neighbor Editing and

Condensing Techniques. Nearest Neighbor Revisited. Condensing Techniques. Proximity Graphs and Decision Boundaries. Editing Techniques . Organization. Last updated: . Nov. . 7, . 2013. Nearest Neighbour Rule.