Parallel Processing

sp2016
lec#1
Dr M Shamim Baig

1.1

Course Information
Prereq:

Computer Architecture, Programming in C/C++

Text/Ref:

- Parallel Computers: Architecture & Programming by Rajaraman

- Introduction to Parallel Computing (2 nd ed) by Grama
- Programming Massively Parallel Processors (2 nd Ed) by Kirk

Grading Policy

Quizzes
: 10 %
Assignments/Project :
Midterm Exam
:
Final Exam
:

15 %
30 %
45 %

Instructor: Dr M Shamim Baig

Email:
msbaig@case.edu.pk
TA: Umer Farooq
- Email: umer.farooq@case.edu.pk Cell# 0345-5896935

Student Resource on CASE network:

Spring2016_ParallelProcessing

1.2

Course Introduction
With advances in computer architecture, high
performance multiprocessor computers have become
readily available & affordable. As a result, high
performance & supercomputing is accessible to a
large segment of industry that was once restricted to
military research & large corporations. The course is
comprised of architecture, algorithms & programming
of multicore & Parallel computing systems. It focuses
on design concepts, principles, paradigms, models,
performance evaluation and real life applications.

1.3

Parallel Computing
Architectures
Current computer architectures are
increasingly relying upon parallelism to
improve performance:
Pipelined processing
Multiple execution units
Multi-core processor (Cores)
Many-core processor (GPU)
Mutiprocessors (CPUs)&
Multicomputers (PCs/ Workstations)
1.4

Course Contents
Parallel Architecture Models:
Implicit: Pipelining, Superscalar & VLIW.
Explicit: SIMD,MIMD, multicore SMP,
vector/array processor, MPP & Clusters

Parallel Memory Models:

SM, DM, DSM, UMA / NUMA
Cache Coherence protocols

Interconnection Networks (IN):

Static vs Dynamic network topologies
LANs vs SANs (Myrinet, Infiniband)

1.5

Course Contentscontd
Collective Communication Operations:
Broadcast, Reduction, Scatter & Gather

Embedding / Mapping application to IN

e.g

Ring/ Mesh into Hypercube & vice versa

Parallel Data Partitioning Techniques

1D/ 2D/ 3D; row-wise/ column-wise;

Parallel Algorithms Design, Analysis &

Programming models: SPMD/MPMD & PRAM
1.6

Course Contentscontd
Multicore SMP programming using OpenMP
Message Passing multicomputer/ Cluster
programming using MPI
Massively (data) Parallel Processing (MPP)
using

GPU / CUDA platform

Heterogeneous Parallel Computing using

CPU & GPU (MPI/openMP/cudaC) together
1.7

Course Contentscontd
Parallel Performance Evaluation
Parallel Linpack standard test criteria
Compute/ Communicate Time Analysis
Systems monitoring/ measurement tools
Parallel Performance Laws
Amdhls Law & Gustafsons Law

1.8

Applications of HPC

1.9

Uses for Parallel Computing:

Historically, parallel computing has been considered to be "the high end of

computing", and has been used to model difficult scientific and engineering
problems found in the real world.
Some examples:
Atmosphere, Earth, Environment
Physics - applied, nuclear, particle, fusion, photonics
Bioscience, Biotechnology, Genetics
Chemistry, Molecular Sciences , Geology, Seismology
Mechanical Engineering - from prosthetics to spacecraft
Electrical Engineering - Circuit Design, Microelectronics
Computer Science & Mathematics

1.10

The Universe is Parallel:

Parallel computing is an evolution of serial computing that
attempts to emulate what has always been the state of affairs
in the natural world:
many complex, interrelated events happening at the same
time, yet within a sequence.
For example:
Galaxy formation
Planetary movement
Weather and ocean
patterns
Tectonic plate drift
Rush hour traffic
Automobile assembly
line
Building space shuttle
Ordering a hamburger
1.11

Uses for Parallel Computing:

Today, commercial applications provide an equal or greater driving force in the development of faster
computers.
These applications require
processing of large amount of data
in sophisticated ways.

For example:

Databases, data mining

Web search engines,
Cloud computing
Adv graphics & virtual reality,
esp entertainment industry
Oil exploration
Medical imaging and diagnosis
Pharmaceutical design
Management of national &
multi-national corporations
Financial & economic modeling
Networked video & multi-media
technologies
Collaborative work environment
Computational Fluid Dynamics
DNA research

1.12

Who and What ?

Top500.org provides statistics on parallel computing users in the charts below

Some things to note: Sectors may overlap

for example, research may be classified research. Respondents have to choose between the two.
"Not Specified" is by far the largest application - probably means multiple applications.

1.13

von Neumann Architecture

Comprised of four
main components:
Memory
Control Unit
Arithmetic Logic
Unit
Input/Output

1.14

serial computing
15

Traditionally, software has been written for serial

computation:
To be run on a single computer having a single CPU
A problem is broken into a discrete series of
instructions.
Instructions are executed one after another.
Only one instruction may execute at any moment in
time.

1.15

parallel computing Problem

The computational problem usually
demonstrates characteristics such as the
ability to be:
Broken apart into discrete pieces of work that
can be solved simultaneously;
Execute multiple program instructions at any
moment in time;
Solved in less time with multiple compute
resources than with a single compute resource.
1.16

parallel computing
In the simplest sense, parallel computing is the simultaneous
use of multiple compute resources to solve a computational
problem:
To be run using multiple CPUs
A problem is broken into discrete parts that can be solved
concurrently
Each part is further broken down into a series of instructions
Instructions from each part execute simultaneously on
different CPUs

17

1.17

Why Use Parallel

Computing?
Limits to serial computing:
Both physical & practical reasons pose
significant constraints to simply building
ever faster serial computers (Moors Law):
Limits to miniaturization
Transmission speeds
Power Dissipation
Energy consumption
Economic limitations
1.18