2012/09/25

Manabu Higashida
manabu@cmc.osaka-u.ac.jp
Host PC: Xeon E3-1225v2/C216


OSOS
High Performance Computing with MPI
Data-Intensive Computing with Hadoop

Windows
/
2
Parallels Workstation 6 Extreme
Windows 7 Enterprise SP1
Data-Intensive
Applications
Zero Client
USB Graphics
USB 3.0
InniBand QDR
The Ideal Entry-level Server
Intel Xeon processor-based servers are a smart investment. They can be used for secure data storage, file sharing,
print and web services, and collaborative applicationsall capabilities that can make your business more efficient and
responsive. A server protects your business-critical information and operations, and enables you to operate more
effectively with existing resources. An Intel Xeon processor E3 family-based server can give you a strategic advantage,
helping you increase sales and improve margins by providing faster access to information and improving workflows
throughout your business.
Compared to a desktop computer, a server based on the Intel Xeon
processor E3-1200 v2 product family is engineered and validated to
run 24/7 and allows you to protect, manage, process, and distribute
your business data more effectively and reliably. With the performance
and capacity of an Intel Xeon processor-based server, you can:
Establish a centralized database for your customer, product, and
financial information. Youll be able to manage vital data more effi-
ciently and effectively to ensure it is accurate, up-to-date, protected,
and accessible to all authorized users anytime and fromanywhere.
Create an in-house network for e-mail, shared calendars,
conferencing, and online document sharing to improve
communication and collaboration among your employees.
Host a corporate web site or improve performance and uptime for
your existing web site, so you can connect more effectively
with your customers and extend the reach of your business.
Take advantage of cloud services to extend your IT capabilities,
while keeping sensitive data and applications on-premise where
you have the control you need to ensure performance, reliability,
security, and compliance.
SMBus
PCIe* x1
8 Lanes
16 GB/s
60 Mb/s
each
4 Lanes
4 GB/s
Intel Gigabit LAN Connect
PCI Express 3.0
PCI Express 2.0
8 Lanes
16 GB/s
PCI Express* 3.0
14 Hi-Speed USB 2.0 Ports;
Dual EHCI; USB Dual Disable
Intel Display Port 1.1
DDR3-1600 ECC 12.8 GB/s
DDR3-1600 ECC 12.8 GB/s
1 GB/s
each x1
Optional
Intel Rapid Storage
Technology 11.x
Up to 6 Serial ATA Ports;
2 @ 6 Gb/s; 4 @ 3 Gb/s
Intel High Denition Audio
8 PCI Express 2.0 x1
DMI 4 GB/s
Intel Flexible
Display Interface
SPI
Intel Xeon
Processor
E3-1200 v2
Product Family
Intel ME Firmware
(NM/AMT) and BIOS Support
Features may vary depending on processor and chipset SKUs.
Intel C202/
204/206/216
Chipset
Intel Integrated
10/100/1000
MAC
2
SSD 120GB
(SATA 6Gbps)
ExpEther
(PCI Bridge+Ethernet)
Native Windows
Applications
HPC
Applications
ExpEther
(PCI Bridge+Ethernet)
Intel MPI
CentOS 6.x
P
a
s
s
-
T
h
r
u

PCI Express over Ethernet (1Gbps)
Cloudera Hadoop
Java VM
CentOS 6.x
40Gbps
Other Hosts
PC
3
SSD 120GB
6Gbps
ExpEther
1GbE
Xeon
Ivy Bridge
3.2GHz
InniBand
QDR

ExpEtherPCI ExpressEthernet
PC


USB2.x/3.0I/O

DVD
USB (CALL)
(DVI/HDMI)
4

5

6
NEC Confidential NEC Corporation 2011 Page 3
@++>I7 Q4++>I7
O~07T~$a>Q7~F
(j@D~I; 149

Express5800/53Xh
@CMC 1F @Q
@D~I 4
Express 5800/R120b-1
@CMC 1F @Q
O~07T~$a>Q7~F
(j@D~I; 272
Express5800/53Xh
j@D~I7II~$
iStorage Ne3-20
30TB
j@D~I7II~$
iStorage Ne3-20
40TB
@D~I 7
Express 5800/R120b-1
Q4@@@@B1F
@@++>I7
O~07T~$a>Q7~F
(j@D~I; 149

Express5800/53Xh
@@@4FD~IQ
@D~I 4
Express 5800/R120b-1
@@@4FD~IQ
j@D~I7II~$
iStorage Ne3-20
20TB
@QQ@@D~7$7TI
7IO>7$7TI
3~D
O0(>
7O>I3>FD~I
$a7Q/
ID~I (@)
ID~I ()
SX-8R
SX-9
66.2Top/s, 5.0TB, 67.4TB
7

1
169 268 138 575
CPU
102.4Gop/s@3.2GHz
169 268 138 575
58.9Top/s
115.2Gop/s@3.6GHz
(TurboBoost)
66.2Top/s
(TurboBoost)

16GB@25.6GB/s 70 - - 70 1.1TB
5.0TB
8GB@12.8GB/s 99 268 138 505 3.9TB
SSD 120GB@6Gbps 169 268 138 575 67.4TB
OS
8
HPC Hadoop
Large Small Server Client
50 525 20 555
2 2 1

28.8Gop/s (*) - -

1.44Top/s 15.12Top/s - -
16.56TFlop/s -

12GB (25.6GB/s) 4GB (12.8GB/s) 12GB 2GB

0.59TB 2.05TB 0.23TB 1.08TB
2.64TB 1.32TB
*)OSIntelAVX(AdvancedVectorExtensions)
1: Windows Terminal Service
Windows 7


USB
Blue-Ray
10




GSE2

CMC2

CMC34
CALL

5
9
2: High-Performance Computing
10

Intel Compiler (C/C++, Fortran) 10.1
Intel MPI 4.0
Intel MKL 10.0

HyperThreading
8GB (12.8GB/s)
8GB+8GB=16GB (25.6GB/s)

OSIntel AVX (Advanced
Vector Extensions)

InniBand


HPC
Large Small
50 525
2

28.8Gop/s (*)

1.44Top/s 15.12Top/s
16.56TFlop/s

12GB (25.6GB/s) 4GB (12.8GB/s)

0.59TB 2.05TB
2.64TB
3: Data-Intensive Computing
Computational Science and Engineering
Data-Intensive Computing
Extreme Science and Engineering Discovery Environment
a.k.a. XSEDE on the U.S. (formerly called TeraGrid)
Big Data
Google BigTable
NoSQL (Not Only SQL)
DFS (Distributed File System) KVS (Key-Value Store)
DBMS
MapReduce
Hadoop
Yahoo!BigTable (Google MapReduce
+ HBaseBigTable)
ApacheOpen SourceFacebook

Amazon Elastic MapReduceIaaS
Cloudera

KerberosDFS
11
Hadoop
Server Client
20 555
2 1

- -

- -
-

12GB 2GB

0.23TB 1.08TB
1.32TB
MPI vs. MapReduce on caliculation of
MPI:
MapReduce:
12
1
x
2
+1
dx =

4
0
1

- {| 4/6 -
/,1 98304 )):]|N.1..



. E})/

]{,,). t [..|..|. t jN"
.


1
2
0
1
1 4
dx
x
t
=
+
}



|:, tan x u = 1
2
1
cos
dx
du u
= , : 0 1 x /: : 0
4
t
u .
1
4 4 2
2 2 2 2
0 0 0
1 1 1 1
cos
1 tan 1 cos cos 4
dx d d
x
t t
t
u u u
u u u
= = =
+ +
} } }

'[]{N"|.

{lN{N".
1
2
0
1
1
dx
x +
}
:
F|
2
1
1
y
x
=
+
x |:| 0 1 x s s
;{.,l[ 0 1 x s s n
`}/ n [,){{)|.
2
1
1
y
x
=
+
: 0 x > [|).
[,){:
2 2 2
1 1
1 1 1
1
n n
k k
n
n n k k
n
= =
=
+
| |
+
|
\ .


,'.,(}.[,).F|d.
.
1 1
2 2 2
0 0
1 1 1
1
n n
k k
n
n n k k
n

= =
=
+
| |
+
|
\ .


/.]
1
2
0
1
4
1
dx
x
t =
+
}
.
1
2 2 2 2
1 0
1 1
4 4
n n
k k
n n
n k n k
t

= =
< <
+ +


/..,/
1
2 2 2 2
0 1
1 1 2
4 4
n n
k k
n n
n k n k n

= =
=
+ +


/,N t |_1..

10 n = ',,`

}
2
1
1
y
x

x
y
2
1
1
y
x

x
y
MPI
#include "mpi.h"!
#include <math.h>!
!
int main(int argc, char *argv[])!
{!
int done = 0, n, myid, numprocs, i;!
double PI25DT = 3.141592653589793238462643;!
double mypi, pi, h, sum, x;!
!
MPI_Init(&argc, &argv);!
MPI_Comm_size(MPI_COMM_WORLD, &numprocs);!
MPI_Comm_rank(MPI_COMM_WORLD, &myid);!
while (!done)!
{!
!if (myid == 0) {!
! printf("Enter the number of intervals: (0 quits) ");!
! scanf("%d", &n);!
!}!
!MPI_Bcast(&n, 1, MPI_INT, 0, MPI_COMM_WORLD);!
!if (n == 0) break;!
!
!h = 1.0 / (double) n;!
!sum = 0.0;!
!for (i = myid + 1; i <= n; i += numprocs) {!
! x = h * ((double)i - 0.5);!
! sum += 4.0 / (1.0 + x*x);!
!}!
!mypi = h * sum;!
!
!MPI_Reduce(&mypi, &pi, 1, MPI_DOUBLE, MPI_SUM, 0,!
! ! MPI_COMM_WORLD);!
!
!if (myid == 0)!
! printf("pi is approximately %.16f, Error is %.16f\n",!
! ! pi, fabs(pi - PI25DT));!
}!
MPI_Finalize();!
return 0;!
}!

1) MPI_Init()
2) MPI_Comm_size()
numprocs
3) MPI_Comm_rank()
myid

0
1) n
2) MPI_Bcast()
n
3)
4) MPI_Reduce()
mypipi
5)

mypi

MPI_Finalize()
13
- {| 4/6 -
/,1 98304 )):]|N.1..



. E})/

]{,,). t [..|..|. t jN"
.


1
2
0
1
1 4
dx
x
t
=
+
}



|:, tan x u = 1
2
1
cos
dx
du u
= , : 0 1 x /: : 0
4
t
u .
1
4 4 2
2 2 2 2
0 0 0
1 1 1 1
cos
1 tan 1 cos cos 4
dx d d
x
t t
t
u u u
u u u
= = =
+ +
} } }

'[]{N"|.

{lN{N".
1
2
0
1
1
dx
x +
}
:
F|
2
1
1
y
x
=
+
x |:| 0 1 x s s
;{.,l[ 0 1 x s s n
`}/ n [,){{)|.
2
1
1
y
x
=
+
: 0 x > [|).
[,){:
2 2 2
1 1
1 1 1
1
n n
k k
n
n n k k
n
= =
=
+
| |
+
|
\ .


,'.,(}.[,).F|d.
.
1 1
2 2 2
0 0
1 1 1
1
n n
k k
n
n n k k
n

= =
=
+
| |
+
|
\ .


/.]
1
2
0
1
4
1
dx
x
t =
+
}
.
1
2 2 2 2
1 0
1 1
4 4
n n
k k
n n
n k n k
t

= =
< <
+ +


/..,/
1
2 2 2 2
0 1
1 1 2
4 4
n n
k k
n n
n k n k n

= =
=
+ +


/,N t |_1..

10 n = ',,`

}
2
1
1
y
x

x
y
2
1
1
y
x

x
y
1
x
2
+1
dx =

4
0
1

2
1
3
numprocs=4
14
start
end
MPI_Init()
MPI_Comm_size()
MPI_Comm_rank()
Keyboard Input
MPI_Bcast()
Operations
MPI_Reduce()
Output
MPI_Finalize()
start
end
MPI_Init()
MPI_Comm_size()
MPI_Comm_rank()
MPI_Bcast()
Operations
MPI_Reduce()
MPI_Finalize()
start
end
MPI_Init()
MPI_Comm_size()
MPI_Comm_rank()
MPI_Bcast()
Operations
MPI_Reduce()
MPI_Finalize()
start
end
MPI_Init()
MPI_Comm_size()
MPI_Comm_rank()
MPI_Bcast()
Operations
MPI_Reduce()
MPI_Finalize()
rank=0 rank=1 rank=2 rank=3
Hadoop
1 of 4
package org.apache.hadoop.examples;!
!
import java.io.IOException;!
import java.math.BigDecimal;!
import java.util.Iterator;!
!
import org.apache.hadoop.conf.Configured;!
import org.apache.hadoop.fs.FileSystem;!
import org.apache.hadoop.fs.Path;!
import org.apache.hadoop.io.BooleanWritable;!
import org.apache.hadoop.io.LongWritable;!
import org.apache.hadoop.io.SequenceFile;!
import org.apache.hadoop.io.Writable;!
import org.apache.hadoop.io.WritableComparable;!
import org.apache.hadoop.io.SequenceFile.CompressionType;!
import org.apache.hadoop.mapred.FileInputFormat;!
import org.apache.hadoop.mapred.FileOutputFormat;!
import org.apache.hadoop.mapred.JobClient;!
import org.apache.hadoop.mapred.JobConf;!
import org.apache.hadoop.mapred.MapReduceBase;!
import org.apache.hadoop.mapred.Mapper;!
import org.apache.hadoop.mapred.OutputCollector;!
import org.apache.hadoop.mapred.Reducer;!
import org.apache.hadoop.mapred.Reporter;!
import org.apache.hadoop.mapred.SequenceFileInputFormat;!
import org.apache.hadoop.mapred.SequenceFileOutputFormat;!
import org.apache.hadoop.util.Tool;!
import org.apache.hadoop.util.ToolRunner;!
!
/**!
* A Map-reduce program to estimate the value of Pi!
* using quasi-Monte Carlo method.!
*!
* Mapper:!
* Generate points in a unit square!
* and then count points inside/outside of the inscribed circle of the square.!
*!
* Reducer:!
* Accumulate points inside/outside results from the mappers.!
*!
* Let numTotal = numInside + numOutside.!
* The fraction numInside/numTotal is a rational approximation of!
* the value (Area of the circle)/(Area of the square),!
* where the area of the inscribed circle is Pi/4!
* and the area of unit square is 1.!
* Then, Pi is estimated value to be 4(numInside/numTotal). !
*/!
!
public class PiEstimator extends Configured implements Tool {!
/** tmp directory for input/output */!
static private final Path TMP_DIR = new Path(!
PiEstimator.class.getSimpleName() + "_TMP_3_141592654");!
!
/** 2-dimensional Halton sequence {H(i)},!
* where H(i) is a 2-dimensional point and i >= 1 is the index.!
* Halton sequence is used to generate sample points for Pi estimation. !
*/!
private static class HaltonSequence {!
/** Bases */!
static final int[] P = {2, 3}; !
/** Maximum number of digits allowed */!
static final int[] K = {63, 40}; !
!
private long index;!
private double[] x;!
private double[][] q;!
private int[][] d;!
!
/** Initialize to H(startindex),!
* so the sequence begins with H(startindex+1).!
*/!
HaltonSequence(long startindex) {!
index = startindex;!
x = new double[K.length];!
q = new double[K.length][];!
d = new int[K.length][];!
for(int i = 0; i < K.length; i++) {!
q[i] = new double[K[i]];!
d[i] = new int[K[i]];!
}!
!
for(int i = 0; i < K.length; i++) {!
long k = index;!
x[i] = 0;!
!
for(int j = 0; j < K[i]; j++) {!
q[i][j] = (j == 0? 1.0: q[i][j-1])/P[i];!
d[i][j] = (int)(k % P[i]);!
k = (k - d[i][j])/P[i];!
x[i] += d[i][j] * q[i][j];!
}!
}!
}!
!
/** Compute next point.!
* Assume the current point is H(index).!
* Compute H(index+1).!
* !
* @return a 2-dimensional point with coordinates in [0,1)^2!
*/!
double[] nextPoint() {!
index++;!
for(int i = 0; i < K.length; i++) {!
for(int j = 0; j < K[i]; j++) {!
d[i][j]++;!
x[i] += q[i][j];!
if (d[i][j] < P[i]) {!
break;!
}!
d[i][j] = 0;!
x[i] -= (j == 0? 1.0: q[i][j-1]);!
}!
}!
return x;!
}!
}!
!

15
Hadoop
2 of 4
/**!
* Mapper class for Pi estimation.!
* Generate points in a unit square!
* and then count points inside/outside of the inscribed circle of the square.!
*/!
public static class PiMapper extends MapReduceBase!
implements Mapper<LongWritable, LongWritable, BooleanWritable, LongWritable>
{!
!
/** Map method.!
* @param offset samples starting from the (offset+1)th sample.!
* @param size the number of samples for this map!
* @param out output {ture->numInside, false->numOutside}!
* @param reporter!
*/!
public void map(LongWritable offset,!
LongWritable size,!
OutputCollector<BooleanWritable, LongWritable> out,!
Reporter reporter) throws IOException {!
!
final HaltonSequence haltonsequence = new HaltonSequence(offset.get());!
long numInside = 0L;!
long numOutside = 0L;!
!
for(long i = 0; i < size.get(); ) {!
//generate points in a unit square!
final double[] point = haltonsequence.nextPoint();!
!
//count points inside/outside of the inscribed circle of the square!
final double x = point[0] - 0.5;!
final double y = point[1] - 0.5;!
if (x*x + y*y > 0.25) {!
numOutside++;!
} else {!
numInside++;!
}!
!
//report status!
i++;!
if (i % 1000 == 0) {!
reporter.setStatus("Generated " + i + " samples.");!
}!
}!
!
//output map results!
out.collect(new BooleanWritable(true), new LongWritable(numInside));!
out.collect(new BooleanWritable(false), new LongWritable(numOutside));!
}!
}!
!
!

/**!
* Reducer class for Pi estimation.!
* Accumulate points inside/outside results from the mappers.!
*/!
public static class PiReducer extends MapReduceBase!
implements Reducer<BooleanWritable, LongWritable, WritableComparable<?>,
Writable> {!
!
private long numInside = 0;!
private long numOutside = 0;!
private JobConf conf; //configuration for accessing the file system!
!
/** Store job configuration. */!
@Override!
public void configure(JobConf job) {!
conf = job;!
}!
!
/**!
* Accumulate number of points inside/outside results from the mappers.!
* @param isInside Is the points inside? !
* @param values An iterator to a list of point counts!
* @param output dummy, not used here.!
* @param reporter!
*/!
public void reduce(BooleanWritable isInside,!
Iterator<LongWritable> values,!
OutputCollector<WritableComparable<?>, Writable> output,!
Reporter reporter) throws IOException {!
if (isInside.get()) {!
for(; values.hasNext(); numInside += values.next().get());!
} else {!
for(; values.hasNext(); numOutside += values.next().get());!
}!
}!
/**!
* Reduce task done, write output to a file.!
*/!
@Override!
public void close() throws IOException {!
//write output to a file!
Path outDir = new Path(TMP_DIR, "out");!
Path outFile = new Path(outDir, "reduce-out");!
FileSystem fileSys = FileSystem.get(conf);!
SequenceFile.Writer writer = SequenceFile.createWriter(fileSys, conf,!
outFile, LongWritable.class, LongWritable.class, !
CompressionType.NONE);!
writer.append(new LongWritable(numInside), new LongWritable(numOutside));!
writer.close();!
}!
}!
!
!

16
Hadoop
3 of 4
/**!
* Run a map/reduce job for estimating Pi.!
*!
* @return the estimated value of Pi!
*/!
public static BigDecimal estimate(int numMaps, long numPoints, JobConf jobConf!
) throws IOException {!
//setup job conf!
jobConf.setJobName(PiEstimator.class.getSimpleName());!
!
jobConf.setInputFormat(SequenceFileInputFormat.class);!
!
jobConf.setOutputKeyClass(BooleanWritable.class);!
jobConf.setOutputValueClass(LongWritable.class);!
jobConf.setOutputFormat(SequenceFileOutputFormat.class);!
!
jobConf.setMapperClass(PiMapper.class);!
jobConf.setNumMapTasks(numMaps);!
!
jobConf.setReducerClass(PiReducer.class);!
jobConf.setNumReduceTasks(1);!
!
// turn off speculative execution, because DFS doesn't handle!
// multiple writers to the same file.!
jobConf.setSpeculativeExecution(false);!
!
//setup input/output directories!
final Path inDir = new Path(TMP_DIR, "in");!
final Path outDir = new Path(TMP_DIR, "out");!
FileInputFormat.setInputPaths(jobConf, inDir);!
FileOutputFormat.setOutputPath(jobConf, outDir);!
!
final FileSystem fs = FileSystem.get(jobConf);!
if (fs.exists(TMP_DIR)) {!
throw new IOException("Tmp directory " + fs.makeQualified(TMP_DIR)!
+ " already exists. Please remove it first.");!
}!
if (!fs.mkdirs(inDir)) {!
throw new IOException("Cannot create input directory " + inDir);!
}!
!

try {!
//generate an input file for each map task!
for(int i=0; i < numMaps; ++i) {!
final Path file = new Path(inDir, "part"+i);!
final LongWritable offset = new LongWritable(i * numPoints);!
final LongWritable size = new LongWritable(numPoints);!
final SequenceFile.Writer writer = SequenceFile.createWriter(!
fs, jobConf, file,!
LongWritable.class, LongWritable.class, CompressionType.NONE);!
try {!
writer.append(offset, size);!
} finally {!
writer.close();!
}!
System.out.println("Wrote input for Map #"+i);!
}!
!
//start a map/reduce job!
System.out.println("Starting Job");!
final long startTime = System.currentTimeMillis();!
JobClient.runJob(jobConf);!
final double duration = (System.currentTimeMillis() - startTime)/1000.0;!
System.out.println("Job Finished in " + duration + " seconds");!
!
//read outputs!
Path inFile = new Path(outDir, "reduce-out");!
LongWritable numInside = new LongWritable();!
LongWritable numOutside = new LongWritable();!
SequenceFile.Reader reader = new SequenceFile.Reader(fs, inFile, jobConf);!
try {!
reader.next(numInside, numOutside);!
} finally {!
reader.close();!
}!
!
//compute estimated value!
return BigDecimal.valueOf(4).setScale(20)!
.multiply(BigDecimal.valueOf(numInside.get()))!
.divide(BigDecimal.valueOf(numMaps))!
.divide(BigDecimal.valueOf(numPoints));!
} finally {!
fs.delete(TMP_DIR, true);!
}!
}
17
Hadoop
4 of 4
/**!
* Parse arguments and then runs a map/reduce job.!
* Print output in standard out.!
* !
* @return a non-zero if there is an error. Otherwise, return 0. !
*/!
public int run(String[] args) throws Exception {!
if (args.length != 2) {!
System.err.println("Usage: "+getClass().getName()+" <nMaps> <nSamples>");!
ToolRunner.printGenericCommandUsage(System.err);!
return -1;!
}!
!
final int nMaps = Integer.parseInt(args[0]);!
final long nSamples = Long.parseLong(args[1]);!
!
System.out.println("Number of Maps = " + nMaps);!
System.out.println("Samples per Map = " + nSamples);!
!
final JobConf jobConf = new JobConf(getConf(), getClass());!
System.out.println("Estimated value of Pi is "!
+ estimate(nMaps, nSamples, jobConf));!
return 0;!
}!
!
/**!
* main method for running it as a stand alone command. !
*/!
public static void main(String[] argv) throws Exception {!
System.exit(ToolRunner.run(null, new PiEstimator(), argv));!
}!
}
18
PiEstimator.java
http://grepcode.com/le/repository.cloudera.com/content/repositories/releases/com.cloudera.hadoop/hadoop-examples/0.20.2-320/org/apache/hadoop/examples/PiEstimator.java
package org.apache.hadoop.examples;!
!
import ...!
!
public class PiEstimator extends Configured implements Tool {!
...!
private static class HaltonSequence {!
... !
double[] nextPoint() { ... }!
}!
!
public static class PiMapper extends MapReduceBase!
implements Mapper<LongWritable, LongWritable, BooleanWritable, LongWritable>
{!
public void map(LongWritable offset,!
LongWritable size,!
OutputCollector<BooleanWritable, LongWritable> out,!
Reporter reporter) throws IOException {!
final HaltonSequence haltonsequence = new HaltonSequence(offset.get());!
...!
for(long i = 0; i < size.get(); ) {!
final double[] point = haltonsequence.nextPoint();!
final double x = point[0] - 0.5;!
final double y = point[1] - 0.5;!
if (x*x + y*y > 0.25) {!
numOutside++;!
} else {!
numInside++;!
}!
...!
out.collect(newBooleanWritable(true),newLongWritable(numInside));!
out.collect(newBooleanWritable(false),newLongWritable(numOutside));!
}!
}!
!
public static class PiReducer extends MapReduceBase!
implements Reducer<BooleanWritable, LongWritable, WritableComparable<?>,
Writable> {!
public void reduce(BooleanWritable isInside,!
Iterator<LongWritable> values,!
OutputCollector<WritableComparable<?>, Writable> output,!
Reporter reporter) throws IOException {!
if (isInside.get()) {!
for(; values.hasNext(); numInside += values.next().get());!
} else {!
for(; values.hasNext(); numOutside += values.next().get());!
}!
}!
!
public void close() throws IOException { ... }!
}!
!
!
public static BigDecimal estimate(int numMaps, long numPoints, JobConf jobConf!
) throws IOException {!
...!
try {!
for(int i=0; i < numMaps; ++i) {!
...!
final LongWritable offset = new LongWritable(i * numPoints);!
final LongWritable size = new LongWritable(numPoints);!
final SequenceFile.Writer writer = SequenceFile.createWriter( ... );!
try {!
writer.append(offset, size);!
} finally { ... }!
...!
} !
...!
!
JobClient.runJob(jobConf);!
...!
LongWritable numInside = new LongWritable();!
LongWritable numOutside = new LongWritable();!
SequenceFile.Reader reader = new SequenceFile.Reader(fs, inFile, jobConf);!
try {!
reader.next(numInside, numOutside);!
} finally {!
reader.close();!
}!
return BigDecimal.valueOf(4).setScale(20)!
.multiply(BigDecimal.valueOf(numInside.get()))!
.divide(BigDecimal.valueOf(numMaps))!
.divide(BigDecimal.valueOf(numPoints));!
}!
}!
!
public int run(String[] args) throws Exception {!
...!
System.out.println("Estimated value of Pi is "!
+ estimate(numMaps, numPoints, jobConf));!
...!
}!
!
public static void main(String[] argv) throws Exception {!
System.exit(ToolRunner.run(null, new PiEstimator(), argv));!
}!
}!
19
size size size
offset offset
Hadooop MapReduce
ResourceManager
Hadoop (HDFS)
(Halton)
numMaps numPoints!
NodeManager
numMapsMapperoffset
size (= numPoints)
numPoints

(numInside++) (numOutside+
+) Reducer
() Reducer
20
Hadoop Slave
NodeManager
DataNode
Hadoop Master
ResourceManager
NameNode
Job
Hadoop Slave
NodeManager
DataNode
Hadoop Slave
NodeManager
DataNode
Task Task Task
Mapper Mapper Mapper
Combiner Combiner Combiner
Practitioner Practitioner Practitioner
Reducer Reducer
Shufe and Sort
2
1 3
3
3
Result
PiEstimator.java: Job Management
public static BigDecimal estimate(int numMaps, long numPoints, JobConf jobConf!
) throws IOException {!
...!
try {!
for(int i=0; i < numMaps; ++i) {!
...!
final LongWritable offset = new LongWritable(i * numPoints);!
final LongWritable size = new LongWritable(numPoints);!
final SequenceFile.Writer writer = SequenceFile.createWriter( ... );!
try {!
writer.append(offset, size);!
} finally { ... }!
...!
} !
...!
!
JobClient.runJob(jobConf);!
...!
LongWritable numInside = new LongWritable();!
LongWritable numOutside = new LongWritable();!
SequenceFile.Reader reader = new SequenceFile.Reader(fs, inFile, jobConf);!
try {!
reader.next(numInside, numOutside);!
} finally {!
reader.close();!
}!
return BigDecimal.valueOf(4).setScale(20)!
.multiply(BigDecimal.valueOf(numInside.get()))!
.divide(BigDecimal.valueOf(numMaps))!
.divide(BigDecimal.valueOf(numPoints));!
}!
}!
!
public int run(String[] args) throws Exception {!
...!
System.out.println("Estimated value of Pi is "!
+ estimate(numMaps, numPoints, jobConf));!
...!
}!
!
public static void main(String[] argv) throws Exception {!
System.exit(ToolRunner.run(null, new PiEstimator(), argv));!
}!
}!
21
b
c
b
a
d
d
a
c
PiEstimator.java: Mapper
public static class PiMapper extends MapReduceBase!
implements Mapper<LongWritable, LongWritable, BooleanWritable, LongWritable> {!
public void map(LongWritable offset,!
LongWritable size,!
OutputCollector<BooleanWritable, LongWritable> out,!
Reporter reporter) throws IOException {!
final HaltonSequence haltonsequence = new HaltonSequence(offset.get());!
...!
for(long i = 0; i < size.get(); ) {!
final double[] point = haltonsequence.nextPoint();!
final double x = point[0] - 0.5;!
final double y = point[1] - 0.5;!
if (x*x + y*y > 0.25) {!
numOutside++;!
} else {!
numInside++;!
}!
...!
out.collect(newBooleanWritable(true),newLongWritable(numInside));!
out.collect(newBooleanWritable(false),newLongWritable(numOutside));!
}!
}!
22
PiEstimator.java: Reducer
public static class PiReducer extends MapReduceBase!
implements Reducer<BooleanWritable, LongWritable, WritableComparable<?>, Writable> {!
public void reduce(BooleanWritable isInside,!
Iterator<LongWritable> values,!
OutputCollector<WritableComparable<?>, Writable> output,!
Reporter reporter) throws IOException {!
if (isInside.get()) {!
for(; values.hasNext(); numInside += values.next().get());!
} else {!
for(; values.hasNext(); numOutside += values.next().get());!
}!
}!
!
public void close() throws IOException { ... }!
}!
!
23
Hadoop: BigTable as a Data-Sink
24
DataStrean0>`
Q 55pj
55j
55lmporL
HDFS(File)System)
Table
HBase(NoSQLDataBase)
FlumeNG)
'/)
Text)
File
jO/@
7I)
Q


[0aI70FlumeNG)AgentI7,pj@1J55]jHDFS^5
?I#5777,)
)
|)%I,[Q55pjI7n)7Jd,[QI7p,
[/@.;10J#Ipj7,)
J,)LJpjj70.;1I7,)
)
>)ume?ng)avro?client)))?H))a?I(?p)aI?F)/@.;1)
)
|)%Ip,
#I[05.1I/L75?I.;1055import7,)
J,)LJimportj70.;1I7,)
])
))>)hadoop)jar)/usr/lib/hbase/hbase.jar)imporIsv)\)
!!!!!!'?DimporIsv.separator=E]'))\)
)))))))?DimporIsv.columns=HBASE_ROW_KEY,|('family:qualier)\)
))))))))5.1(.;1()
)
|)%I,|,)
J,[Q55pj|7]JU55j70|0,=]5
5n|j@1HbaseI71n,|I7KEY|
'family:qualier)n[LI@1Jd,@55n|j@1I
t,
Text)
File
Text)
File
Text)
File
CSV#5)
CharacterSet=uZ8
FlumeNG)
Agent)
FlumeNG)
AVRO?Client)
FlumeNG)
AVRO?Client)
FlumeNG)
AVRO?Client)
imporIsv))
HPC
PCIntel

Intel MPI 3.2
http://www.hpc.cmc.osaka-u.ac.jp/j/tebiki/
pcc-impi.html!
PC OpenMP
http://www.hpc.cmc.osaka-u.ac.jp/j/tebiki/
pcc_omp.html!

(Intel Compiler 10.1, MPI 4.0 Update 2)
:
C Compiler
/sc/cmc/apl/intel/impi/4.0.2.003/bin64/mpiicc!
C++ Compiler
/sc/cmc/apl/intel/impi/4.0.2.003/bin64/mpiicpc
Fortran Compiler
/sc/cmc/apl/intel/impi/4.0.2.003/bin64/mpiifort!

(TB)
: /work/!

SSD (GB)
: /hpc!
MB/s

25

NQS -q
:
: PCC-T (1/16/32/64, 4GB)!
: PCC-S (4/8/16/32, 4GB or 12GB)
: PCC-M (1/4/8, 4GB)!
2 (OpenMP2)
4GB
12GB

128
26
7|7=| j)`L5`?5Jj
(j#`?5J'.O`O/|
2012/08/30
/;v@j
LxecuLloo Oueue
1oLal
676oodec
Oueue PL Oueue PL Oueue PL
N128Ly - N128cu - N128 - =
N64Ly 4 266
NJ2Ly 8 NJ2cu 4 J84
N16Ly 16 N16cu 8 J84
N8Lcu 4
N8cu 8 N8mo 8 192
N4Lcu 4
N4cu 8 N4mo J2 192
N1Ly J2 N1mo 64 96
6X,UE7PCO?5||PCC)0JCPUt707I,7It7,
|QI]7,UE,@I),(ILy]7),cuUE),mo)|D7,
7IJ`?I,[,7|I7I[4CB>,2CPU7,
QI2I}7#@|,#Q0I)[,
Loyooaka culLa mlooH
268oodec 148oodec 1J8oodec
7|7=| j)`L5`?5Jj
(j#`?5J'.O`O/|
2012/08/30
PouLlog Oueue
N1Ly N16Ly NJ2Ly N64Ly
N4cu N8cu N16cu NJ2cu
N1mo N4mo N8mo
)0|^0jp),
|`.,|70?JI``I/[Lt,
PCC-1
PCC-6
PCC-V
0ecLloaLloo Oueue PouLlogOueue
NQS

-I
,

/sc/cmc, /sc/ile,/sc/
rcnp
/work/


/
1) HOME indata le1
/work//input

HOME $HOME short
(/sc/cmc/short/)
!!
!#PBS -I $HOME/indata/file1,input/!
2) /sc/cmc/short//indata
/work//input

!
!#PBS -I /sc/cmc/short//indata/,input/


-O
,


/
1) /work//output
data HOME output
data
HOME $HOME short
(/sc/cmc/short/)

#PBS -I $HOME/output/data,output/data!

2) /work//output
/sc/cmc/short//output

!
!#PBS -I /sc/cmc/short//output/,output/
27

28
NEC Confidential NEC Corporation 2011 Page 3
@++>I7 Q4++>I7
O~07T~$a>Q7~F
(j@D~I; 149

Express5800/53Xh
@CMC 1F @Q
@D~I 4
Express 5800/R120b-1
@CMC 1F @Q
O~07T~$a>Q7~F
(j@D~I; 272
Express5800/53Xh
j@D~I7II~$
iStorage Ne3-20
30TB
j@D~I7II~$
iStorage Ne3-20
40TB
@D~I 7
Express 5800/R120b-1
Q4@@@@B1F
@@++>I7
O~07T~$a>Q7~F
(j@D~I; 149

Express5800/53Xh
@@@4FD~IQ
@D~I 4
Express 5800/R120b-1
@@@4FD~IQ
j@D~I7II~$
iStorage Ne3-20
20TB
@QQ@@D~7$7TI
7IO>7$7TI
3~D
O0(>
7O>I3>FD~I
$a7Q/
ID~I (@)
ID~I ()
SX-8R
SX-9
Intel MPI Benchmark
QDR InniBand
: 1.1 (Native) 1.9 (Guest w/Pass-Thru)
: 3GB/s
29
0
1
10
100
1,000
10,000
1
.
0
0
E
+
0
0

1
.
0
0
E
+
0
1

1
.
0
0
E
+
0
2

1
.
0
0
E
+
0
3

1
.
0
0
E
+
0
4

1
.
0
0
E
+
0
5

1
.
0
0
E
+
0
6

1
.
0
0
E
+
0
7

M
b
y
t
e
s
/
s
e
c

#bytes
Native
Guest w/Pass-Thru

HPC
ParallelsAVX

Hadoop
1



30