Mr. Scan: Efficient Clustering with - PowerPoint Presentation

1. Mr. Scan: Efficient Clustering with MRNet and GPUsEvan Samanas and Ben Welton

2. Density-based clusteringDiscovers the number of clustersFinds oddly-shaped clusters2Mr. Scan: Efficient Clustering with MRNet and GPUs

3. Goal: Find regions that meet minimum density and spatial distance characteristicsThe two parameters that determine if a point is in a cluster is Epsilon (Eps), and MinPtsIf the number of points in Eps is > MinPts, the point is a core point.For every discovered point, this same calculation is performed until the cluster is fully expandedClustering Example (DBSCAN[1])3Mr. Scan: Efficient Clustering with MRNet and GPUsEpsMinPtsMinPts: 3[1] M. Ester et. al., A density-based algorithm for discovering clusters in large spatial databases with noise, (1996)

4. Scaling DBSCANPDBSCAN (1999)[2]Quality equivalent to single DBSCANLinear speedup up to 8 nodes DBDC (2004)[3]Sacrifices quality~30x speedup on 15 nodesPDSDBSCAN (2012) [4]Quality equivalent to single node DBSCAN5675x Speedup on 8192 nodes (72 Million Points) 2 Map/Reduce attempts (2011, 2012)Quality equivalent to single node DBSCAN6x speedup on 12 nodes4Mr. Scan: Efficient Clustering with MRNet and GPUs[2] X. Xu et. al., A fast Parallel Clustering Algorithm for Large Spatial Databases (1999)[3] E. Januzaj et. al., DBDC: Density Based Distributed Clustering (2004)[4] M Patwary et. al., A new scalable parallel DBSCAN algorithm using the disjoint-set data structure (2012)

5. Challenges of scaling DBSCANData distributionHow do we effectively take an input file and create partitions that can be clustered by DBSCAN?Distributed 2-D partitioner reading from a distributed file systemLoad balancingHow to keep variance in clustering times across nodes to a minimum?Dense Box MergeHow do we reduce the amount of data needed for the merge while keeping accuracy high?Representative points 5Mr. Scan: Efficient Clustering with MRNet and GPUs

6. 6Mr. Scan: Efficient Clustering with MRNet and GPUsMRNet – Multicast / Reduction NetworkGeneral-purpose TBON API Network: user-defined topology Stream: logical data channelto a set of back-endsmulticast, gather, and custom reduction Packet: collection of data Filter: stream data operatorsynchronizationtransformationWidely adopted by HPC toolsCEPBA toolkit Cray ATP & CCDBOpen|SpeedShop & CBTFSTATTAUFE………BEappappappappBEappappappappBEappappappappBEappappappappCPCPCPCPCPCPF(x1,…,xn)

7. TBON Computation7Mr. Scan: Efficient Clustering with MRNet and GPUsFEBEappappappappBEappappappappBEappappappappCPCPBEappappappappIdeal Characteristics: Filter output size constant or decreasing Computation rate similar across levels Adjustable for load balanceData Size: 10MB per BEPacket Size: ≤10 MBPacket Size:≤10 MB~10 sec~40 sec…4x~10 sec~10 sec~10 secTotal Time: ~30 secTotal Time: ~60 sec

8. Intro to Mr. Scan8Mr. Scan: Efficient Clustering with MRNet and GPUsBEBEBECPCPBEDBSCANMergeFEMr. Scan PhasesPartition: DistributedDBSCAN: GPU(@ BE)Merge: CPU (x #levels)Sweep: CPU (x #levels)FEBEBEBEBEMergeFSSweepSweep

9. Mr. Scan Architecture9Mr. Scan: Efficient Clustering with MRNet and GPUsTime: 0 Time: 18.2 Min PartitionerDBSCANMerge & SweepClustering 6.5 Billion PointsFS Read 224 SecsFS Write489 SecsMRNetStartup130 SecsFS Read: 24 SecsDBSCAN168 SecsMerge Time: 6 SecsSweep Time: 4 SecsWrite Output: 19 Secs

10. Partition PhaseGoal: Partitions computationally equivalent to DBSCANAlgorithm:Form initial partitionsAdd shadow regionsRebalance10Mr. Scan: Efficient Clustering with MRNet and GPUs

11. Distributed Partitioner11Mr. Scan: Efficient Clustering with MRNet and GPUs

12. GPU DBSCAN Filter 12Mr. Scan: Efficient Clustering with MRNet and GPUsDBSCAN is performed in two distinct stepsStep 1: Detect Core PointsBlock 1Block 2Block 900T1T2T512T1T2T512T1T2T512Block 1T1T2T512Block 2T1T2T512Block 900T1T2T512Step 2: Expand core points and color

13. Dense Box 13Mr. Scan: Efficient Clustering with MRNet and GPUsOne significant scalability issue is dealing with dense regions of dataDensity increases the computation cost of DBSCANR2 Requires morecomparison operationsR1R2We reduce the computation cost of high density regions by pre-clustering these regionsKD-TreeLook at each leaf bounding box looking for boxes with point count > minpts and size < 0.35 * epsDBSCAN no longer needs to expand these regions`

14. Merge AlgorithmMerge overlapping clusters found on different nodes. Two steps in the merge operationSelect Representative points (BE)Merge operation14Mr. Scan: Efficient Clustering with MRNet and GPUs

15. Representative PointsThese are points that represent the core points in the dataset. Create a boundary which at least one core point shared between overlapping clusters must be contained.15Mr. Scan: Efficient Clustering with MRNet and GPUsRepresentative points are the points closest to the corners and middle of the side of the eps boxThese points create a boundary (shaded region) which a point must fall in to merge overlapping clusters

16. Merge Algorithm16Mr. Scan: Efficient Clustering with MRNet and GPUsMerge algorithm is responsible for merging overlapping clusters detected on different DBSCAN nodes. Need to handle the merge with low overhead and without the full datasetNode 1Node 2Core PointNon-Core Point1. Core/Core overlapCore Point in common. 64 operations to detect.Node 1Node 2Core PointNon-Core Point2. Non-core/Core overlapCore point seen as non-core by one node. MinPts * 2 operations required to detect

17. Sweep StepGet cluster identifiers and file offsets down to BE’s to write final clusters. FE gives each cluster a unique ID and a file offset. This data is passed back down to the BE that holds the data in the cluster.Data is written out to disk by the BE.17Mr. Scan: Efficient Clustering with MRNet and GPUs

18. Experiment SetupDataset: Generated data with distribution from real Twitter dataMeasuring:Weak Scaling up to 8192 GPUsStrong ScalingQuality compared to single-threaded DBSCAN18Mr. Scan: Efficient Clustering with MRNet and GPUs

19. Results 19Mr. Scan: Efficient Clustering with MRNet and GPUsWeak Scaling: 4096x data/compute increase 18.48x-31.68x time increase

20. Results Breakdown – Partition Phase@ 6.5 Billion Points: 65.9% of Mr. Scan’s time 94.6% I/O time 20Mr. Scan: Efficient Clustering with MRNet and GPUs

21. Results Breakdown – GPU Cluster Time21Mr. Scan: Efficient Clustering with MRNet and GPUs

22. Strong Scaling22Mr. Scan: Efficient Clustering with MRNet and GPUs

23. Quality23Mr. Scan: Efficient Clustering with MRNet and GPUs

24. Future WorkRemove partitioner’s I/O bottleneckMultiple dimensions24Mr. Scan: Efficient Clustering with MRNet and GPUs

25. ConclusionClustered 6.5 billion points with DBSCAN in 18.2 minutesControlled computational variance of DBSCANPartitioner I/O = scaling enemy25Mr. Scan: Efficient Clustering with MRNet and GPUs

26. Questions?26A Brief Discussion of Ways and Means

27. Summary of previous Mr. Scan implementation27Mr. Scan: Efficient Clustering with MRNet and GPUsFEBEBEBECPCPBEDBSCANAlgorithm StepsSpatialDecomp: CPU(@ FE)DBSCAN: CPU or GPU(@ BE)DrawBoundBox: CPU or GPUMergeCluster: CPU (x #levels)MergeCluster