Map Reduce Algorithm Design

Fundamental Research Group,IIT Bombay

Designing MapReduce Algorithms

Anurag Sharma
Fundamental Research Group IIT Bombay

Anurag Sharma (IIT Bombay)

1/5

Map Reduce Algorithm Design

Fundamental Research Group,IIT Bombay

Programmer and Hadoop

Out of Control of a programmer Where a mapper or reducer runs (i.e., on which node in the cluster) When a mapper or reducer begins or nishes Which input key-value pairs are processed by a specic mapper Which intermediate key-value pairs are processed by a specic reducer In Control of the programmer Construct complex data structures as keys and values Execute user-specied initialization code at the beginning/end of a map or reduce task Preserve state in both mappers and reducers Control the sort order of intermediate keys Control the partitioning of the key space
Anurag Sharma (IIT Bombay) 2/5

Map Reduce Algorithm Design

Fundamental Research Group,IIT Bombay

Local Aggregation
Reduction of data transfer over network or to the disk in a cluster based system Introducing the use of Combiner & In-Mapper Combine
combiners are mini reducers

Anurag Sharma (IIT Bombay)

3/5

Map Reduce Algorithm Design

Fundamental Research Group,IIT Bombay

Pairs and Stripes

building a word co-occurence system. Pairs

Approach

Anurag Sharma (IIT Bombay)

4/5

Map Reduce Algorithm Design

Fundamental Research Group,IIT Bombay

fragile
Anurag Sharma (IIT Bombay) 5/5

Map Reduce Algorithm Design

Fundamental Research Group,IIT Bombay

References I

Data-Intensive Text Processing with MapReduce. 5

Anurag Sharma (IIT Bombay)

5/5

Map Reduce Algorithm Design

Fundamental Research Group,IIT Bombay

Anurag Sharma (IIT Bombay)

5/5

Map Reduce Algorithm Design

Fundamental Research Group,IIT Bombay

5thebibliography

Anurag Sharma (IIT Bombay)

5/5