Designing MapReduce Algorithms - PowerPoint Presentation

Slide1

Designing MapReduce Algorithms

Ch. 3 Lin and Dyer’s text

http://lintool.github.io/MapReduceAlgorithms/MapReduce-book-final.pdf

Pages 43-73 (39-69)Slide2

Word count:

Local aggregation as opposed to external combiner that is NOT guaranteed by the

Hadoop frameworkMay not work all the time: what if wanted word “mean” instead of word “count”: may have to adjust <k,v> types at the output of mapWord co-occurrence (matrix)Very important since many (many) problems are expressed and solved using matricesPairs and stripes approachesAnd comparison of these two methods P.60 (56)

ImprovementsSlide3

First version simplistic counts

Then “relative frequency” instead of counts

What is relative frequency? Instead of absolute countsf(wi /wj) = N(wi,wj)/∑w’ (w

i

, w’)For example, if word “dog” co-occurred with “food” 23 times, and “dog” co-occurred with all words 460 times, then relative frequency is 23/460 = 1/20 = 0.05Also the 460 could come from many mappers, many documents over the entire corpus.These co-occurrences from every mapper are delivered to “corresponding reducer” with a special keyThis is delivered as special key item < (wi, *) , count> as the first <k,v> pairThe magic is that reducer processes < (wi, *) , count>

Co-OccurrenceSlide4

Key

Value

Reducer operation/computeResult(dog,*)[200,

350,650

]One per mapper with combiner Marginal∑w’ (wi, w’)=1200(dog, bark)60Relative frequency

<(dog, bark), 0.05>

(dog, cat)12Relative frequency<(dog, cat), 0.01>(dog, food)600Relative frequency<(dog, food), 0.5>….…

At the reducer: Blue: reducer1/Orange: reducer 2

Key

Value

Reducer

operation

Result

(tiger,*)

[100,

300,600]

Compute

marginal

∑

w’

(wi, w’)=1000

(tiger, cub)

10

Compute

R

.freq

<(tiger, cub), 10/1000>

(tiger, hunt)

100

Compute

R.freq

<(tiger,

hunt), 100/1000>

(tiger, prey)

200

Compute

R.freq

<(tiger, prey), 200/1000>

….Slide5

4 different reducersSlide6

Emitting a special key-value pair for each co-occurring word pair in the

mapper to

capture its contribution to the marginal.Controlling the sort order of the intermediate key so that the key-value pairs representing the marginal contributions are processed by the reducer before any of the pairs representing the joint word co-occurrence counts.Defining a custom partitioner to ensure that all pairs with the same left word are shuffled to the same reducer.Preserving state across multiple keys in the reducer to first compute the

marginal based

on the special key-value pairs and then dividing the joint counts by the marginal to arrive at the relative frequencies.RequirementsSlide7

Lets generalize this

<(var34, left), value>

<(var34, right), value><(var34, middle), value> all delivered to the same reducer.. What can you do with this?Reducer can “middle(left’s value, right’s value) “  <var34, computedValue>

Some more:

<KEY complex object, VALUE complex object>You can do anything you want for function… “KEY.operation” on “VALUE.data” Therein lies the power of MR.Slide8

Text word count

Text co-occurrence

 pairs and stripesNumerical data processing with most math functionsHow about sensor data?Consider m sensors, sending out readings rx at various times ty resulting large volume of data of the format:(t1;m1; r80521)(t1;m2; r14209)(t1;m3; r76042):::(t2;m1; r21823)(t2;m2; r66508)

(t2;m3; r98347

)Suppose you wanted to know the readings by the sensors, how could process the above to get that info?Use MR to do that…<m1 , (t1; r80521)> etc.But what if wanted that sorted by time t that is a part of the value?Problem discussed so farSlide9

Solution 1: Let the reducer do the in-memory sorting

 memory bottle neck

Solution 2: Move value to be sorted to the key, and modify the shuffler and partitionerIn the later, the “secondary sorting” is left to the framework and it excels in doing this anyway. So solution 2 is a preferred approach.Lesson: Let the framework do what it is good at and don’t try to move into your code… in the latter you will be regressing to the “usual” coding practices and ensuing disadvantages

In memory sorting vs. value-key conversionSlide10

Reduce-side join is intuitive but inefficient

Map-side join requires simple merge of respective input files and appropriate sort by the MR framework

In-memory joins can be done for smaller data.We will NOT discuss this in detail since there are other solutions such as Hive, Hbase available for warehouse data. We will look into these later.Relational Joins/warehouse data