Parallel Processing

By: Bela Desai Guided by: Anita

Patel
What is parallel processing?

 The simultaneous use of more than one CPU to

execute a program is called parallel processing.

 It works on the principle that large problems can

often be divided into smaller ones, which are then
solved concurrently i.e in parallel.
Serial computation
Parallel Computation
Flynn’s Classification

SISD SIMD
Single Instruction, Single Data Single Instruction, Multiple Data

MISD MIMD
Multiple Instruction, Single Multiple Instruction, Multiple
Data Data
Single Instruction, Single Data (SISD):

 A serial (non-parallel) computer.

 Single instruction.
 Single data.
 Examples: most PCs, single CPU
workstations and mainframes.
Single Instruction, Multiple Data
(SIMD):
Multiple Instruction, Single Data
(MISD):
Multiple Instruction, Multiple Data
(MIMD):
Types Of Parallel Processing

 Bit-level parallelism
 Instruction-level parallelism
 Data parallelism
 Task parallelism
Bit-level parallelism

 Speed-up in computer architecture was driven by

doubling computer word size—the amount of
information the processor can execute per cycle.
 Increasing the word size reduces the number of
instructions the processor must execute to perform
an operation on variables whose sizes are greater
than the length of the word.
Instruction-level parallelism
 The instructions can be re-ordered and combined into
groups which are then executed in parallel without
changing the result.
 Processor with an N-stage pipeline can have up to N
different instructions at different stages of completion.
 The canonical example of a pipelined processor is a
RISC processor, with five stages: instruction fetch,
decode, execute, memory access, and write back. The
Pentium 4 processor had a 35-stage pipeline.
Data parallelism
 1: PREV2 := 0
 2: PREV1 := 1
 3: CUR := 1
 4: do:
 5: CUR := PREV1 + PREV2
 6: PREV2 := PREV1
 7: PREV1 := CUR
 8: while (CUR < 10)
Task parallelism
 Entirely different calculations can be performed on either
the same or different sets of data.
 This contrasts with data parallelism, where the same
calculation is performed on the same or different sets of
data. Task parallelism does not usually scale with the size
of a problem.
Shared Memory
 Multiple processors can operate independently but
share the same memory resources.

 Changes in a memory location effected by one

processor are visible to all other processors.

 Shared memory machines can be divided into two

main classes based upon memory access times: UMA
and NUMA
Uniform Memory Access (UMA):

 Most commonly represented today by Symmetric

Multiprocessor (SMP) machines.

 Identical processors.

 Equal access and access times to memory.

 Sometimes called CC-UMA - Cache Coherent UMA.

Non-Uniform Memory Access
(NUMA):
 Often made by physically linking two or more SMPs.
 One SMP can directly access memory of another
SMP.
 Not all processors have equal access time to all
memories.
 Memory access across link is slower
 If cache coherency is maintained, then may also be
called CC-NUMA - Cache Coherent NUMA
Distributed Memory
Hybrid Distributed-Shared Memory
Parallel Programming Models

 Shared Memory Model

 Thread Model
 Message Passing Model
 Data Parallel Model
 Hybrid
Shared memory model
 In the shared-memory programming model, tasks share a
common address space, which they read and write
asynchronously.
 An advantage of this model from the programmer's point of
view is that the notion of data "ownership" is lacking, so there
is no need to specify explicitly the communication of data
between tasks. Program development can often be simplified.
 An important disadvantage in terms of performance is that it
becomes more difficult to understand and manage data
locality.
Thread Model
POSIX thread
Library based; requires parallel coding
 Specified by the IEEE POSIX 1003.1c standard
(1995).
 C Language only
 Commonly referred to as Pthreads.
 Most hardware vendors now offer Pthreads in addition
to their proprietary threads implementations.
 Very explicit parallelism; requires significant
programmer attention to detail.
OpenMP
 Compiler directive based; can use serial code
 Jointly defined and endorsed by a group of major
computer hardware and software vendors. The
OpenMP Fortran API was released October 28, 1997.
The C/C++ API was released in late 1998.
 Portable / multi-platform, including Unix and Windows
NT platforms
 Available in C/C++ and Fortran implementations
 Can be very easy and simple to use - provides for
"incremental parallelism"
Message Passing Model
 A set of tasks that use their own local memory during
computation. Multiple tasks can reside on the same
physical machine as well across an arbitrary number
of machines.
 Tasks exchange data through communications by

sending and receiving messages.

Data transfer usually requires cooperative operations
to be performed by each process. For example, a send
operation must have a matching receive operation.
Message Passing Model
Data Parallel Model
 Most of the parallel work focuses on performing
operations on a data set. The data set is typically
organized into a common structure, such as an array
or cube.
 A set of tasks work collectively on the same data
structure, however, each task works on a different
partition of the same data structure.
 Tasks perform the same operation on their partition of
work, for example, "add 4 to every array element".
Data Parallel Model
Hybrid
1. Single Program Multiple Data (SPMD)

2.Multiple Program Multiple Data (MPMD)

Conclusion
With the help of parallel processing, highly complicated
scientific problems that are otherwise extremely difficult
to solve can be solved effectively. Parallel computing can
be effectively used for tasks that involve a large number
of calculations, have time constraints and can be divided
into a number of smaller tasks.