PPT-Twister: A Runtime for Iterative MapReduce

Jaliya Ekanayake Community Grids Laboratory Digital Science Center Pervasive Technology Institute Indiana University HPDC 2010 MAPREDUCE10 Workshop Chicago 06222010 Acknowledgements to. S. A. L. S. A. . HPC Group . http://. salsahpc.indiana.edu. School of Informatics and Computing. Indiana University. Judy . Qiu. . Thilina. . Gunarathne. . CAREER Award. Outline. Iterative . and . Hadoop. Debapriyo Majumdar. Data Mining – Fall 2014. Indian Statistical Institute Kolkata. November 10, 2014. Let’s keep the intro short. Modern data mining: process immense amount of data quickly. TexPoint fonts used in EMF. . Read the TexPoint manual before you delete this box.: . A. A. A. A. A. A. A. A. A. A. Iterative methods for . Ax=b. . Iterative methods produce a sequence of approximations. , Collective Communication, and Services. Oral Exam, . Bingjing. Zhang. Outline. MapReduce. MapReduce. Frameworks. Iterative . MapReduce. Frameworks. Frameworks Based on . MapReduce. and Alternatives. Efficient and scalable architectures to perform pleasingly parallel, MapReduce and iterative data intensive computations on cloud environments. Thilina. . Gunarathne. (tgunarat@indiana.edu). Advisor : . CloudCom. 2010. Indianapolis, Indiana, USA. Nov 30 – Dec 3, 2010. Azure . MapReduce. AzureMapReduce. A . MapRedue. runtime for Microsoft Azure using Azure cloud services. Azure Compute. Azure BLOB storage for in/out/intermediate data storage. Data Enabled . S. cience. S. A. L. S. A. . HPC Group . http://. salsahpc.indiana.edu. School of Informatics and Computing. Indiana University. Bioinformatics Pipeline. Gene Sequences (N = 1 Million). Efficient and scalable architectures to perform pleasingly parallel, MapReduce and iterative data intensive computations on cloud environments. Thilina. . Gunarathne. (tgunarat@indiana.edu). Advisor : . . Shantenu Jha represented by . Geoffrey Fox. gcf@indiana.edu. . . http://www.infomall.org. . http://www.futuregrid.org. . . http://salsahpc.indiana.edu/. . Director, Digital Science Center, Pervasive Technology Institute. Mega 10^6. Giga 10^9. Tera. 10^12. Peta. 10^15. Pig Latin. Understanding . MapReduce. . in Context. Linux HPC. Bare-system. Amazon Cloud. Windows Server HPC. Bare-system. Virtualization. Cross Platform Iterative . Judy Qiu, Indiana University. Prof. David Crandall. Computer Vision. Prof. Filippo Menczer. Complex Networks and Systems. Bingjing. Zhang. Acknowledgement. Xiaoming. . Gao. Stephen Wu. Thilina. . S. cience. S. A. L. S. A. . HPC Group . http://. salsahpc.indiana.edu. School of Informatics and Computing. Indiana University. Bioinformatics Pipeline. Gene Sequences (N = 1 Million). Distance Matrix. Computations. K-means. Performance of K-Means. Smith Waterman is a non iterative case and of course runs fine. Matrix Multiplication . 64 cores. Square blocks Twister. Row/Col . decomp. Twister. ”. Cathy O’Neil & Rachel . Schutt. , 2013. R & Hadoop. Compute squares. 2. R. # create a list of 10 integers. ints. <- 1:10. # equivalent to . ints. <- c(1,2,3,4,5,6,7,8,9,10). # compute the squares.