Pig : Building High-Level

Dataflows over Map-Reduce

Utkarsh Srivastava

Research &
Cloud Computing
 Data Processing Renaissance

Internet companies swimming in data

• E.g. TBs/day at Yahoo!

Data analysis is “inner loop” of product innovation

Data analysts are skilled programmers
 Data Warehousing …?

Scale Often not scalable enough

Prohibitively expensive at web scale

$$$$
• Up to $200K/TB

• Little control over execution method

SQL • Query optimization is hard
• Parallel environment
• Little or no statistics
• Lots of UDFs
 New Systems For Data Analysis

Map-Reduce

Apache Hadoop ...

Dryad
 Map-Reduce

Input k1 v1 k1 v1 Output
records k2 v2 records
map k1 v3 reduce
k1 v3 k1 v5

map
k2 v4 k2 v2 reduce
k1 v5 k2 v4

Just a group-by-aggregate?
 The Map-Reduce Appeal

Scalable due to simpler design

Scale • Only parallelizable operations
• No transactions

$ Runs on cheap commodity hardware

SQL Procedural Control- a processing “pipe”

 Disadvantages

1. Extremely rigid data flow M R

Other flows constantly hacked in

M M R M

Join, Union Split Chains

2. Common operations must be coded by hand

• Join, filter, projection, aggregates, sorting, distinct
3. Semantics hidden inside map-reduce functions
• Difficult to maintain, extend, and optimize
 Pros And Cons

Need a high-level, general data flow language

 Enter Pig Latin

Need a high-level, general data flow language

 Outline

• Map-Reduce and the need for Pig Latin

• Pig Latin

• Compilation into Map-Reduce

• Example Generation

• Future Work
 Example Data Analysis Task
Find the top 10 most visited pages in each category

Visits Url Info

User Url Time Url Category PageRank

Amy cnn.com 8:00 cnn.com News 0.9

Amy bbc.com 10:00 bbc.com News 0.8

Amy flickr.com 10:05 flickr.com Photos 0.7

Fred cnn.com 12:00 espn.com Sports 0.9

 Data Flow
Load Visits

Group by url

Foreach url
Load Url Info
generate count

Join on url

Group by category

Foreach category
generate top10 urls
 In Pig Latin
visits = load ‘/data/visits’ as (user, url, time);
gVisits = group visits by url;
visitCounts = foreach gVisits generate url, count(visits);

urlInfo = load ‘/data/urlInfo’ as (url, category, pRank);

visitCounts = join visitCounts by url, urlInfo by url;

gCategories = group visitCounts by category;

topUrls = foreach gCategories generate top(visitCounts,10);

store topUrls into ‘/data/topUrls’;

 Step-by-step Procedural Control
Target users are entrenched procedural programmers
The step-by-step method of creating a program in Pig is much cleaner and
simpler to use than the single block method of SQL. It is easier to keep track of
what your variables are, and where you are in the process of analyzing your
data.
Jasmine Novak
Engineer, Yahoo!

With the various interleaved clauses in SQL, it is difficult to know what is

actually happening sequentially. With Pig, the data nesting and the temporary
tables get abstracted away. Pig has fewer primitives than SQL does, but it’s
more powerful.
David Ciemiewicz
Search Excellence, Yahoo!

• Automatic query optimization is hard

• Pig Latin does not preclude optimization
 Quick Start and Interoperability
visits = load ‘/data/visits’ as (user, url, time);
gVisits = group visits by url;
visitCounts = foreach gVisits generate url, count(urlVisits);

urlInfo = load ‘/data/urlInfo’ as (url, category, pRank);

visitCounts = join visitCounts by url, urlInfo by url;

gCategories = group visitCounts by category;
topUrls = foreach gCategories
Operates generate
directly overtop(visitCounts,10);
files

store topUrls into ‘/data/topUrls’;

 Quick Start and Interoperability
visits = load ‘/data/visits’ as (user, url, time);
gVisits = group visits by url;
visitCounts = foreach gVisits generate url, count(urlVisits);

urlInfo = load ‘/data/urlInfo’ as (url, category, pRank);

visitCounts = join visitCounts by url, urlInfo by url;

gCategories = group visitCounts by category;
Schemasgenerate
topUrls = foreach gCategories optional; top(visitCounts,10);
Can be assigned dynamically
store topUrls into ‘/data/topUrls’;
 User-Code as a First-Class Citizen
visits = load ‘/data/visits’ as (user, url, time);
gVisits = group visits functions
User-defined by url; (UDFs)
visitCountscan
= foreach gVisits
be used generate
in every url, count(urlVisits);
construct
• Load, Store
urlInfo =• load ‘/data/urlInfo’
Group, as (url, category, pRank);
Filter, Foreach

visitCounts = join visitCounts by url, urlInfo by url;

gCategories = group visitCounts by category;
topUrls = foreach gCategories generate top(visitCounts,10);

store topUrls into ‘/data/topUrls’;

 Nested Data Model
• Pig Latin has a fully-nestable data model with:
–Atomic values, tuples, bags (lists), and maps

finance
yahoo , email
news

• More natural to programmers than flat tuples

• Avoids expensive joins
 Nested Data Model
Decouples grouping as an independent operation
User Url Time group Visits
Amy cnn.com 8:00 group by url cnn.com
Amy cnn.com 8:00

Amy bbc.com 10:00 Fred cnn.com 12:00

Amy bbc.com 10:05

Fred cnn.com 12:00 Amy bbc.com 10:00
bbc.com
Amy bbc.com 10:05
• Common case: aggregation on these nested sets
• Power users:
I franklysophisticated UDFs,than
like pig much better e.g.,
SQLsequence
in some analysis
respects (group + optional flatten works better for me,
• EfficientI love
Implementation (see paper)
nested data structures).”

Ted Dunning
Chief Scientist, Veoh
19
CoGroup
results revenue
query url rank query adSlot amount
Lakers nba.com 1 Lakers top 50
Lakers espn.com 2 Lakers side 20
Kings nhl.com 1 Kings top 30
Kings nba.com 2 Kings side 10

group results revenue

Lakers nba.com 1 Lakers top 50
Lakers
Lakers espn.com 2 Lakers side 20

Kings nhl.com 1 Kings top 30

Kings
Kings nba.com 2 Kings side 10

Cross-product of the 2 bags would give natural join

 Outline

• Map-Reduce and the need for Pig Latin

• Pig Latin

• Compilation into Map-Reduce

• Example Generation

• Future Work
 Implementation

SQL user

automatic
rewrite + Pig or
optimize Pig is open-source.
or

http://hadoop.apache.org/pig
Hadoop
Map-Reduce

cluster • ~50% of Hadoop jobs at

Yahoo! are Pig
• 1000s of jobs per day
 Compilation into Map-Reduce
Map1 Every group or join operation
Load Visits
forms a map-reduce boundary
Group by url
Reduce1
Map2
Foreach url
Load Url Info
generate count

Join on url
Reduce2
Map3
Other operations Group by category
pipelined into map and Reduce3
Foreach category
reduce phases generate top10(urls)
 Optimizations: Using the Combiner

Input k1 v1 k1 v1 Output
records k2 v2 records
map k1 v3 reduce
k1 v3 k1 v5

map
k2 v4 k2 v2 reduce
k1 v5 k2 v4

Can pre-process data on the map-side to reduce data shipped

• Algebraic Aggregation Functions
• Distinct processing
 Optimizations: Skew Join
• Default join method is symmetric hash join.
cross product carried out on 1 reducer

group results revenue

Lakers nba.com 1 Lakers top 50
Lakers
Lakers espn.com 2 Lakers side 20

Kings nhl.com 1 Kings top 30

Kings
Kings nba.com 2 Kings side 10

• Problem if too many values with same key

• Skew join samples data to find frequent values
• Further splits them among reducers
 Optimizations: Fragment-Replicate Join

• Symmetric-hash join repartitions both inputs

• If size(data set 1) >> size(data set 2)

– Just replicate data set 2 to all partitions of data set 1

• Translates to map-only job

– Open data set 2 as “side file”
 Optimizations: Merge Join

• Exploit data sets are already sorted.

• Again, a map-only job

– Open other data set as “side file”
Optimizations: Multiple Data Flows
Load Users Map1

Filter bots

Group by Group by
state demographic
Reduce1

Apply udfs Apply udfs

Store into ‘bystate’ Store into ‘bydemo’

Optimizations: Multiple Data Flows
Load Users Map1

Filter bots

Split

Group by Group by
state demographic

Demultiplex Reduce1

Apply udfs Apply udfs

Store into ‘bystate’ Store into ‘bydemo’

 Other Optimizations

• Carry data as byte arrays as far as possible

• Using binary comparator for sorting

• “Streaming” data through external executables

Performance
 Outline

• Map-Reduce and the need for Pig Latin

• Pig Latin

• Compilation into Map-Reduce

• Example Generation

• Future Work
 Example Dataflow Program
LOAD LOAD
(user, url) (url, pagerank)

JOIN
on url
Find users that
FOREACH GROUP tend to visit
user, canonicalize(url) on user
high-pagerank
FOREACH pages
user, AVG(pagerank)

FILTER
avgPR> 0.5
 Iterative Process
LOAD LOAD
(user, url) (url, pagerank)

JOIN Joining on right

on url attribute?

FOREACH GROUP
user, canonicalize(url) on user

Bug in UDF FOREACH

canonicalize? user, AVG(pagerank)
Everything being
FILTER filtered out?
avgPR> 0.5

No Output ☹
 How to do test runs?

• Run with real data

– Too inefficient (TBs of data)

• Create smaller data sets (e.g., by sampling)

– Empty results due to joins [Chaudhuri et. al. 99], and
selective filters

• Biased sampling for joins

– Indexes not always present
 Examples to Illustrate Program
LOAD LOAD (www.cnn.com, 0.9)
(www.frogs.com, 0.3)
(user, url) (url, pagerank) (www.snails.com, 0.4)

(Amy, cnn.com)
(Amy, http://www.frogs.com)
(Fred, www.snails.com/index.html)
JOIN (Amy, www.cnn.com, 0.9)
(Amy, www.frogs.com, 0.3)
on url
(Fred, www.snails.com, 0.4)

FOREACH GROUP
user, canonicalize(url) on user (Amy, www.cnn.com, 0.9)
( Amy, )
(Amy, www.frogs.com, 0.3)

FOREACH ( Fred, (Fred, www.snails.com, 0.4) )

(Amy, www.cnn.com) user, AVG(pagerank)

(Amy, www.frogs.com)
(Amy, 0.6)
(Fred, www.snails.com)
(Fred, 0.4)
FILTER
avgPR> 0.5
(Amy, 0.6)
 Value Addition From Examples

• Examples can be used for

– Debugging
– Understanding a program written by someone else
– Learning a new operator, or language
 Good Examples: Consistency
LOAD LOAD
(user, url) (url, pagerank)

(Amy, cnn.com)
(Amy, http://www.frogs.com) JOIN
(Fred, www.snails.com/index.html)
on url

FOREACH GROUP
user, canonicalize(url) on user 0. Consistency

FOREACH
(Amy, www.cnn.com) user, AVG(pagerank) output example
(Amy, www.frogs.com) =
(Fred, www.snails.com)
operator applied on input example
FILTER
avgPR> 0.5
 Good Examples: Realism
LOAD LOAD
(user, url) (url, pagerank)

(Amy, cnn.com)
(Amy, http://www.frogs.com) JOIN
(Fred, www.snails.com/index.html)
on url

FOREACH GROUP
user, canonicalize(url) on user 1. Realism

FOREACH
(Amy, www.cnn.com) user, AVG(pagerank)
(Amy, www.frogs.com)
(Fred, www.snails.com)
FILTER
avgPR> 0.5
 Good Examples: Completeness
LOAD LOAD
(user, url) (url, pagerank)
2. Completeness

JOIN
on url Demonstrate the salient
properties of each operator,
GROUP e.g., FILTER
FOREACH
user, canonicalize(url) on user

FOREACH
user, AVG(pagerank)
(Amy, 0.6)
(Fred, 0.4)
FILTER
avgPR> 0.5
(Amy, 0.6)
 Good Examples: Conciseness
LOAD LOAD
(user, url) (url, pagerank)
3. Conciseness
(Amy, cnn.com)
(Amy, http://www.frogs.com) JOIN
(Fred, www.snails.com/index.html)
on url

FOREACH GROUP
user, canonicalize(url) on user

FOREACH
(Amy, www.cnn.com) user, AVG(pagerank)
(Amy, www.frogs.com)
(Fred, www.snails.com)
FILTER
avgPR> 0.5
 Implementation Status

• Available as ILLUSTRATE command in open-source release

of Pig

• Available as Eclipse Plugin (PigPen)

• See SIGMOD09 paper for algorithm and experiments

 Related Work
• Sawzall
– Data processing language on top of map-reduce
– Rigid structure of filtering followed by aggregation
• Hive
– SQL-like language on top of Map-Reduce
• DryadLINQ
– SQL-like language on top of Dryad
• Nested data models
– Object-oriented databases
 Future / In-Progress Tasks

• Columnar-storage layer
• Metadata repository
• Profiling and Performance Optimizations
• Tight integration with a scripting language
–Use loops, conditionals, functions of host language
• Memory Management
• Project Suggestions at:
http://wiki.apache.org/pig/ProposedProjects
Credits
 Summary

• Big demand for parallel data processing

– Emerging tools that do not look like SQL DBMS
– Programmers like dataflow pipes over static files

• Hence the excitement about Map-Reduce

• But, Map-Reduce is too low-level and rigid

Pig Latin
Sweet spot between map-reduce and SQL