PPT-MapReduce

and Hadoop Debapriyo Majumdar Data Mining Fall 2014 Indian Statistical Institute Kolkata November 10 2014 Lets keep the intro short Modern data mining process immense amount of data quickly. With . a heavy debt to:. Google Map Reduce OSDI 2004 slides. code.google.com. You are an engineer at:. Hare-brained-scheme.com. Your boss, comes to your office and says:. . “We’re going to be hog-nasty rich!. : Simplified Data Processing on Large Clusters. Jeffrey Dean & . Sanjay . Ghemawat. Appeared in:. OSDI '04: Sixth Symposium on Operating System Design and Implementation, San Francisco, CA, December, 2004. . Based on the text by Jimmy Lin and Chris Dryer; and on the yahoo tutorial on . mapreduce. at . http://developer.yahoo.com/hadoop/tutorial/index.html. Namenode responsibilities:. Namespace management: file name, locations, access privileges etc.. Computations. K-means. Performance of K-Means. Smith Waterman is a non iterative case and of course runs fine. Matrix Multiplication . 64 cores. Square blocks Twister. Row/Col . decomp. Twister. , Collective Communication, and Services. Oral Exam, . Bingjing. Zhang. Outline. MapReduce. MapReduce. Frameworks. Iterative . MapReduce. Frameworks. Frameworks Based on . MapReduce. and Alternatives. Simplified Data Processing on Large . Clusters. by Jeffrey Dean and Sanjay . Ghemawa. Presented by Jon Logan. Outline. Problem Statement / Motivation. An Example Program. MapReduce. . vs. Hadoop. GFS / HDFS. Efficient and scalable architectures to perform pleasingly parallel, MapReduce and iterative data intensive computations on cloud environments. Thilina. . Gunarathne. (tgunarat@indiana.edu). Advisor : . Parallel Computing. MapReduce. Examples. Parallel Efficiency. Assignment. Parallel Computing. Parallel efficiency with . p. processors. Traditional parallel computing:. focus on compute intensive tasks. , the Big Data Workhorse. Vyassa Baratham, Stony Brook University. April 20, 2013, 1:05-2:05pm. cSplash. 2013. Use several computers to process large amounts of data. Often significant . distribution overhead. Yang . Ruan. , . Zhenhua . Guo. , . Yuduo. Zhou, Judy . Qiu. , Geoffrey Fox. Indiana . University, US. Outline. Introduction and Background. MapReduce. Iterative MapReduce. Distributed Workflow . Management Systems. Presented By. Shefali. . Gundecha. Srinivas . Narne. Yash. Kulkarni. Papers to be discussed…. Y. Shan, B. Wang, J. Yan, Y. Wang, N. Xu, and H. Yang, . " FPMR: MapReduce Framework on FPGA: A Case Study of . ”. Cathy O’Neil & Rachel . Schutt. , 2013. R & Hadoop. Compute squares. 2. R. # create a list of 10 integers. ints. <- 1:10. # equivalent to . ints. <- c(1,2,3,4,5,6,7,8,9,10). # compute the squares. Source. MapReduce. : Simplified Data Processing in Large Clusters. . Jefferey. Dean and Sanjay . Ghemawat. . OSDI 2004. Example Scenario. 3. Genome data from roughly . one million users. 125 MB of data per user. MapReduce. Architecture. MapReduce. Internals. MapReduce. Examples. JobTracker. Interface. MapReduce. : A Real World Analogy. Coins Deposit. ?. MapReduce. : A Real World Analogy. Coins Deposit. Coins Counting Machine.