Beruflich Dokumente
Kultur Dokumente
Data Tools
Presen
ted
By
Outline
The world map of big data tools
Layered architecture
Big data tools for HPC and supercomputing
MPI
MapReduce model
Iterative MapReduce model
DAG model
Graph model
Collective model
The
World of
Big Data
Tools
DAG Model
MapReduce Model
Graph Model
Hadoop
MPI
HaLoop
Twister
For
Iterations
/
Learning
Spark
Harp
Dryad/
DryadLI
NQ
For
Query
Giraph
Hama
GraphLab
GraphX
Drill
Stratosphere
Reef
Pig/PigLatin
Hive
Tez
Shark
MRQL
For
Streamin
g
S4
Samza
Storm
Spark
Streaming
BSP/Collective
Model
Message
Message Protocols:
Protocols: Thrift,
Thrift, Protobuf
Protobuf (NA)
(NA)
Distributed
Distributed Coordination:
Coordination: ZooKeeper,
ZooKeeper, JGroups
JGroups
Security
Security &
& Privacy
Privacy
Monitoring:
Monitoring: Ambari,
Ambari, Ganglia,
Ganglia, Nagios,
Nagios, Inca
Inca (NA)
(NA)
Cross Cutting
Capabilities
NA:
NA:
Pegas
Pegas
us,
us,
Kepler
Kepler
,,
Swift,
Swift,
Taver
Taver
na,
na,
Triden
Triden
t,t,
Active
Active
BPEL,
BPEL,
BioKe
BioKe
pler,
pler,
Galax
Galax
yy
NA Non Apache
projects
Oozie,
Oozie,
ODE,
ODE,
Airav
Airav
ata
ata
and
and
OOD
OOD
TT
(Tools
(Tools
))
Orche
Orche
stratio
stratio
nn &
&
Workf
Workf
low
low
Machine Learning
Statistics, Bioinformatics
Imagery
R, Bioconductor (NA)
Linear Algebra
ImageJ (NA)
Hcatalog
Interfaces
Pig
Shark
MRQL
(Procedural
Language)
(SQL on
Spark, NA)
(SQL on Hadoop,
Hama, Spark)
Impala (NA)
Cloudera
(SQL on Hbase)
S4
Samza
Yahoo LinkedIn
Stream
Giraph
~Pregel
Pegasus
on Hadoop
(NA)
Graph
Swazall
(Log Files
Google NA)
In memory distributed databases/caches: GORA (general object from NoSQL), Memcached (NA),
Redis(NA) (key value), Hazelcast (NA), Ehcache (NA);
ORM Object Relational Mapping: Hibernate(NA), OpenJPA and
Extraction Tools
NA Non Apache
projects
Tika
MySQL
(Content)
(NA)
(NA)
Arrays,
Phoenix
(SQL on
HBase)
Message
Message Protocols:
Protocols: Thrift,
Thrift, Protobuf
Protobuf (NA)
(NA)
Security
Security &
& Privacy
Privacy
Distributed
Distributed Coordination:
Coordination: ZooKeeper,
ZooKeeper, JGroups
JGroups
MongoDB
(NA)
R,Python
Lucene
Solr
Berkeley
DB
Accumulo
(Data on
HDFS)
(Data on
HDFS)
Azure
Table
NoSQL: TripleStore
Neo4J
Yarcdata
Java Gnu
(NA)
Commercial
(NA)
Jena
Sesame
(NA)
HBase
(Solr+
Cassandra)
+Document
Cassandra
(DHT)
CouchDB
Data Transport
Solandra
NoSQL: Column
SciDB
NoSQL: Document
Monitoring:
Monitoring: Ambari,
Ambari, Ganglia,
Ganglia, Nagios,
Nagios, Inca
Inca (NA)
(NA)
UIMA
(Entities)
(Watson)
SQL
JDBC Standard
User Level
FUSE(NA)
POSIX Interface
RDF
Dynamo
Amazon
Riak
~Dynamo
File
Management
SparkQL
AllegroGraph
Commercial
Voldemort
~Dynamo
RYA RDF on
Accumulo
iRODS(NA)
Interoperability Layer
Whirr / JClouds
OCCI CDMI (NA)
DevOps/Cloud Deployment
Puppet/Chef/Boto/CloudMesh(NA)
IaaS System Manager Open Source
Commercial Clouds
OpenStack, OpenNebula, Eucalyptus, CloudStack, vCloud, Amazon, Azure, Google
Bare
Metal
parallel processes.
Collective communication operations
Broadcast, Scatter, Gather, Reduce, Allgather, Allreduce, Reduce-
scatter.
Popular implementations
MPICH (2001)
OpenMPI (2004)
http://www.open-mpi.org/
MapReduce Model
Google MapReduce (2004)
Jeffrey Dean et al. MapReduce: Simplified Data Processing on Large
HDFS)
Disk based intermediate data transfer in Shuffling
MapReduce programming model
Computation pattern: Map tasks and Reduce tasks
Data abstraction: KeyValue pairs
Google MapReduce
Mapper: split, read, emit
intermediate KeyValue
pairs
(1) fork
User
Program
Reducer: repartition,
emits final output
(1) fork
(1)
forkMaster (2)
(2)
assig
assign
n
map
Worker
(6) writeOutput
reduc
Worker
e
File 0
Split 0 (3) read
(4) local
Split 1
Worker write
Output
Split 2
Worker
File 1
Worker
(5) remote
read
Input files
Map phase
Intermediate files
(on local disks)
Reduce phase
Output files
Worker Nodes
configureMaps()
Local
Disk
configureReduce(
)
while(condition){
Cacheable map/reduce
tasks
runMapReduce(.
Map
May scatter/broadcast
..)
()
<Key,Value> pairs directly
Reduce
Iteration
May merge data in shuffling
()
s
Combine()
Communications/data transfers
operation
via the pub-sub broker network &
direct TCP
updateCondition(
Main program may contain many
)} //end while
close()
MapReduce invocations or
iterative MapReduce invocations
Model Composition
Apache Spark (2010)
Matei Zaharia et al. Spark: Cluster Computing with Working Sets,.
HotCloud 2010.
Matei Zaharia et al. Resilient Distributed Datasets: A Fault-Tolerant
Abstraction for In-Memory Cluster Computing. NSDI 2012.
http://spark.apache.org/
Resilient Distributed Dataset (RDD)
RDD operations
MapReduce-like parallel operations
Active
Vote to
halt
Superstep as iteration
Superste
p0
Superste
p1
Superste
p2
Superste
p3
Master/worker model
Graph partition: hashing
Fault tolerance: checkpointing and
confined recovery
Maximum Value Example
GraphLab (2010)
Yucheng Low et al. GraphLab: A New Parallel Framework for
Data Graph
Edge-cut (Giraph
model)
Option 2
Collective communication optimization
Not considered by previous BSP model but well developed in MPI
Initial attempts in Twister and Spark on clouds
Mosharaf Chowdhury et al. Managing Data Transfers in Computer Clusters
with Orchestra. SIGCOMM 2011.
Bingjing Zhang, Judy Qiu. High Performance Clustering of Social Images in a
Map-Collective Programming Model. SOCC Poster 2013.
Collective Model
Harp (2013)
https://github.com/jessezbj/harp-project
Hadoop Plugin (on Hadoop 1.2.1 and Hadoop 2.2.0)
Hierarchical data abstraction on arrays, key-values and graphs
Harp Design
Parallelism Model
Map-Collective Model
MapReduce
Model
M
Application
MapReduce
Applications
Map-Collective
Applications
M
M
Shuffle
R
Architecture
M
M
Collective
Communication
Harp
Framework
Resource
Manager
MapReduce V2
YARN
Edge
Table
Array Partition
< Array Type
>
Edge
Partition
Table
Partition
Message
Partition
KeyValue
Partition
Broadcast, Send
Long
Array
Int
Array
Basic Types
Double
Array
Byte
Array
Vertices, Edges,
Messages
Array
Key-Values
Struct Object
Broadcast, Send,
Gather
Commutable
Shantenu Jha et
al. A Tale of Two
Data-Intensive
Paradigms:
Applications,
Abstractions, and
Architectures.
2014.
WDA-MDS Performance on
Big Red II
WDA-MDS
Yang Ruan, Geoffrey Fox. A Robust and Scalable Solution for
Allgather
Bucket algorithm
Allreduce
Bidirectional exchange algorithm
Parallel Efficiency
Based On 8 Nodes and 256 Cores
MLlib on Spark
http://spark.apache.org/mllib/
GraphLab Toolkits
http://graphlab.org/projects/toolkits.html
GraphLab Computer Vision Toolkit
SQL-like Query
Apache Hive (2007)
Reynold Xin et al. Shark: SQL and Rich Analytics at Scale. Technical
http://mrql.incubator.apache.org/
On top of Apache Hadoop, Apache Hama, and Apache Spark
REEF
Retainable Evaluator Execution Framework
http://www.reef-project.org/
Provides system authors with a centralized (pluggable) control
flow
Embeds a user-defined system controller called the Job Driver
Event driven control
Thank You!
Questions?