PPT-Map-Reduce Graph Processing

Adapted from UMD Jimmy Lins slides which is licensed under a Creative Commons AttributionNoncommercialShare Alike 30 United States See http creativecommonsorg licensesby nc . : Simplified Data Processing on Large Clusters. Jeffrey Dean & . Sanjay . Ghemawat. Appeared in:. OSDI '04: Sixth Symposium on Operating System Design and Implementation, San Francisco, CA, December, 2004. . R. e. d. u. c. e. . Simplified Data Processing on Large Clusters. (Without the Agonizing Pain). Presented by Aaron Nathan. The Problem. Massive amounts of data. >100TB (the internet). Needs simple processing. Lecture 9. Outline. Map Reduce, cont.. Index compression. [Amazon Web Services]. Map Reduce. Map: (key, value) . . . list(key. ’, value’). Reduce: (key’, . list(value. ’) . . list (value’). Recap: data-intensive cloud computing. Just database management on the cloud. But scaling it to thousands of nodes. Handling partial failures gracefully. Sacrificing strong ACID. Graph processing frameworks (. Science Applications. http://salsahpc.indiana.edu . School of Informatics and Computing. Indiana University. Application Classes. 1. Synchronous. Lockstep Operation as in SIMD architectures. SIMD. 2. Thilina Gunarathne (. tgunarat@indiana.edu. ). Judy . Qiu (. xqiu@indiana.edu. ). Dennis . Gannon (. dennis.gannon@microsoft.com). Introduction. Three disruptions. Big Data. MapReduce. Cloud Computing. Hao Wei. 1. , . Jeffrey Xu Yu. 1. , Can L. u. 1. , . Xuemin. Lin. 2. . 1 . The . Chinese University of Hong Kong, Hong Kong. 2 . The . University of New South Wales. , . Sydney, Australia. Graph in Big Data . Condie. UC . Berkeley. Slides by . Kaixiang. MO. kxmo@cse.ust.hk. Outline. Background. Motivation: Block . vs. . pipeline. Hadoop. Online Prototype Model. Pipeline within job. Online Aggregation. Continuous Queries. Data-Intensive Distributed Computing Part 4: Analyzing Graphs (2/2) This work is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States See http://creativecommons.org/licenses/by-nc-sa/3.0/us/ for details Big Data Infrastructure Week 5: Analyzing Graphs (2/ 2) This work is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States See http://creativecommons.org/licenses/by-nc-sa/3.0/us/ for details Stijn Eyerman, Wim Heirman, Ibrahim Hur, Joshua B. Fryman. 2. DARPA HIVE project. “To build a graph analytics processor that can process streaming graphs 1000X faster and at much lower power than current processing technology”. Jiaul. Paik. Email:. . jia.paik@gmail.com. Today’s Topics. Map-reduce: Additional Details. Inverted Index using Map-reduce. Introduction to Spark. Spark Demo. Map-reduce Internals: Additional Details. Source. MapReduce. : Simplified Data Processing in Large Clusters. . Jefferey. Dean and Sanjay . Ghemawat. . OSDI 2004. Example Scenario. 3. Genome data from roughly . one million users. 125 MB of data per user. Cast of thousands. Mihai. Pop. Michael Schatz. Dan . Sommer. University of Maryland Center for Computational Biology. Faisal Khan, Ken Hahn UW . David Schwartz, LMCG. In 2003…. http://labs.google.com/papers/gfs.html.