Unit 5

Developing parallel computing Applications

Topics:

1. OpenMP Implementation in C

1.1 Execution model

1.2 Memory Model

2. Directives

2.1. Conditional Compilation

2.2. Internal Control variables

2.3.Parallel Construct

2.4. Work Sharing Constructs

2.5. Combined Parallel Work Sharing Constructs

2.6.Master and Synchronization Constructs (From RTU genius)

3. Run time library routines (from RTU genius)

3.1. Execution Environment Routines

3.2. Lock Routines

3.3. Timing Routines

4. Simple Examples in C (not good)

5. Basics of MPI (From RTU genius)

1. Open MP Implementation in C

• OpenMP is a portable programming interface for shared memory computers.

• OpenMP stands for open specifications for multiprocessing and used by
shared memory and shared address space programming model.
• More than one processor executes concurrently and share a common
available memory, multithreading is also used for the execution of multiple
processes.
• Its major goal is to offer a common specification that will let the programmer
easily design new parallel applications or make some changes in the existing one
to take advantages of shared memory system.
• Its portability is its major aim.
• It can be compiled by any compiler supporting OpenMP.
• One more programming model is distributed memory systems, which is also
called message passing model. In which each processor is having its own private
memory, the memory is not shared among processors.
• OpenMP can be added to serial program to show how the work is shared
among multiple threads executed on multiple processors.
• It is primarily a directive based method to invoke parallel computations on
many shared memory multiprocessor. Example of openMP implementation is
FORTRAN and C++.
• OpenMP constructs are defined by pragmas, derivatives and API calls in the
code.
• OpenMp constructs allow parallelization and synchronization in the
programmer
• It also allows run time environment variables to specify run time
configurations.

Advantageous because:
a) Use structural parallel programming efficiency.
b) Compiler has control so simple to use.
c) Can be run on different domain.

1.1. OpenMP Execution Model

• OpenMP is shared memory parallel programming model. So whole data is

shared among multiple threads and easily accessible to all the threads.
• The data available can be shared or private.
• Shared data can be accessible by all the threads but private data can be
accessed only by the thread that owns it.
• Data transfer is transparent to the programmer and synchronization takes
place implicitly.
• Flush operation is used to synchronize memory, which gives the current value
of shared objects to all threads.
• All variables have a label attached to them.
• If labeled ‘Private’, then the data is visible to only one thread. If there is any
change being made in local data, then it is visible only to themselves not to
others. Example: Local variables in a function that is executed in parallel.
• If labeled ‘Shared’, it is visible to all the threads. If there is any change being
made to data then it is visible to all the threads. Example: global data.
 T
T

Priva
te Priva
Shared memory te

T
T
T

Priva Priva
te te

Priva
te

1.2. OpenMP Execution Model

• Application Programming Interface of openMP is developed to activate

portable shared memory parallel programming.
• To execute a program OS creates a process and resources are allocated to
that process. With the help of multithreading resources are shared among
processes to execute the program. It may be possible that multiple threads on
single processor or threads concurrently on multiple processors run to execute a
program.

Processor Processor Processor Processor

1 2 3 4

Synchronized
processors

Communicatin
Shared memory g
parallel Shared via
programming memory shared
variable

• OpenMP makes use of Fork-Join model for parallel execution.

• OpenMP is basically intended to support the programs that will execute
correctly both as parallel programs and as sequential programs.
• An openMP program written in C/C++ API begins execution as a single thread
of execution called as master thread. The master thread executes in serial until
the first parallel construct is encountered. The master thread creates a team of
threads when the parallel construct is encountered and the master thread really
becomes the master.
• Except for the work sharing construct, each team member executes the
statement in the extent of a parallel region. Work sharing constructs must be
encountered by all the threads in the team in the same order and the
statements within the associated structured block are executed by one or more
of threads.
• If a thread modifies its shared object, it effects its own execution as well as of
other threads. If the object is volatile modification is guaranteed to be complete
after the other threads point of view at the execution point, otherwise it would
be completed after the modifying thread and then the other threads encounter a
“flush” directive that specifies the object.
• Upon Completion of the parallel construct, the threads in the team
synchronize at an implicit barrier and only the master thread continues
execution.
• Any number of parallel constructs can be specified in a single program.
Master
threads
Initial
Fork threads
state
Multiple
Th1 Th2 Th3 Th4
threads
Fork threads End state

• Serial executions of a program are done by master threads. Master thread

controls all the threads so it has the power to create and organize and manage
other threads for parallel operations. It is called fork operation.
• After execution completes the control is again returned to single thread, it is
called Join operation.

2. Directives

• Directives specify the program behavior.

• The syntax of OpenMP directive is #pragma omp directive-name [clause
[[clause], ..] ..], new line
• Directives are defined in #pragma directive defined in the C and C++
standards.
• Compilers supporting openMP and C++ API provide these directives.
Compilers have a command line option. It activates and allows interpretation of
all OpenMP directives.
• ‘#pragma omp’ is used to reduce the potential for conflicts with other
directives with same name in non openMP or others.
• Directives are case sensitive.
• Clauses on directives may be repeated as needed, the order in which clauses
appear in the directive is not important. They can be described according to the
list of description. Only one directive name can be specified per directive,
multiple names are not allowed.

2.1. Conditional Compilation

The OpenMP macro name is defined by OpenMP compliant implementation. This
macro is not the subject of a #define or a #undef preprocessing directive.
Example:
# ifdef_OPENMP
iam= omp_get_thread_num () + index;
# endif

2.2. Internal Control variables

• The internal control variables controlled by the openMP implementation that

govern the behavior of a program at run time.
• They cannot be accessed or modified through OpenMP functions and
environment variables.
1. n threads: Shows the number of threads which are in queue for
execution.
2. dyn_var: It controls the dynamic adjustment of the number of threads to
be used for future parallel regions. It finds whether the dynamic
adjustment is enabled or not.
3. nest_var: It controls whether the nested parallelism is enabled for future
parallel regions.
4. run_sched_var: It stores the scheduling information to be used for loop
regions using schedule clause.
5. nthread_var: It stores the number of threads requested for the
execution of future regions.
6. def_sched_var: It stores implementation defined scheduling information
for parallel region.

Some other directives and clause

1. if Clause:

Syntax-> if (expression)
 If the given expression is true then the process continues otherwise
it is run by only one thread.
2. Num_threads Clause:
Syntax: num_threads(scalar_integer_expression)
It supports the number of threads used in the parallel code of
program. It always gives the integer number.
3. Copying Clause: Copy the private variables of initial thread to the
private variables of other threads copying.
4. Flush Clause: Goal is to make a thread’s a temporary view of
shared data consistent with the values in memory.
#pragma omp flush [(list)]

2.3. Parallel Construct

• Parallel construct is a construct that defines a region of program where
execution takes place in parallel. A construct is a statement that consists of a
directive and the structured block.
• Example: #pragma omp parallel [clause [[,] clause]…] new line
• There are number of clauses as follows:
If (scalar expression)
private (variable list) (data is not shared)
first private (variable list) (priority is given to private data)
default (shared/ none) (by default only shared data is there)
shared (variable list) (data is shared)
copyin (variable list) (copy the data to other threads)
reduction (operator : variable-list)
num threads (integer expression)
• If there is a parallel construct in a thread then some conditions are created
like a team of threads got created with this thread as the member thread in the
condition if one of the following cases are true:
1. Number of clause is/are present
2. The if expression evaluates to a non-zero value
• The master thread is assigned the thread number 0and threads in the team
executed the region in parallel but in the case that if expression evaluates as 0
value, then the execution takes place serially.
• Some rules are to be considered to determine the number of threads that are
requested:
1. If the num_threads clause is present, then the value of integer expression
is the number of threads requested.
2. If the omp_set_num threads library function has been called, then the
value of the argument in the most recently executed call is the number of
threads requested.
3. If the OMP_NUM_THREADS is defined, then the value of this environment
variable is the number of threads requested.
 4. F any of the above rules has not been satisfied then the number of
threads requested is implementation defined.
• The number of threads executing the parallel region also depends upon
dynamic adjustment of the number of threads.
• If the dynamic adjustment is disabled then parallel execution takes place
otherwise the number of threads is the maximum number of threads that can
execute in parallel.
• If the dynamic adjustment is disabled and the request number of threads
exceeds the number that the system can supply, then the behavior of program is
implementation defined and in this case the program may interrupt or serial
execution may take place.
• The ‘omp_set_dynamic’ library functions and ‘omp_dynamic’ environment
variables are used to enable or disable the dynamic adjustment oh the number
of threads.
• If a thread in a team executing in a parallel region encounters another
parallel construct, then it creates a new team and becomes the master of the
new team.
• Nested parallel regions are serialized by default.
• The number of threads created once remains constant for the duration of that
parallel region.
• Two types of libraries are included in parallel construct:
1. omp_get_dynamic which set OMP_DYNAMIC environment variable to true.
This enables dynamic threads.
2. omp_get_nested which set OMP_NESTED environment variable to true. It
enables nested parallel codes.
• Restrictions to the parallel Directive are:
1. At most one ‘if’ clause can appear on the directive.
2. A ‘throw’ execute inside a parallel region must cause execution to resume
within the dynamic extent of the same structures block.
3. Only a single ‘num_threads’ clause can appear on the directive.

2.4. Work Sharing Constructs

• A work sharing construct is that which distributes the execution of the

statement associated to it among the members of the team that has
encountered it.
• These constructs do not launch new threads and there is no implied barrier
on entry to a work sharing construct.
• It only focuses on multiple threads execution not a single thread execution
• OpenMP defined the following work Sharing constructs:
1. For construct
2. Sections construct
3. Single construct
FOR construct
• The for construct acts as an iterative construct and its iteration is
distributed among the threads that already exist in the team executing the
parallel construct to which it binds.
• The for directive holds thefor constructs that specifies that iteration pf
the asosiated loop will execute in parallel.
• Syntax:
#pragma omp for [clause [[,] clause]…] new line
For loop
The clause may be one of the following:
private (variable list)
first private (variable list)
last private (variable list)
reduction (operator: variable list)
ordered
schedule (kind [, chunk_size])
nowait
• Restrictions to the ‘for’ directive are:
1. Must be structured block and its execution must not be terminated
by a break statement.
2. The value of loop control expression of the ‘for’ loop must be same
for all the threads in the team.
3. The for loop iteration variables must have a signed integer type.
4. Only a single schedule, ordered and nowait clause can appear on a
for directive
5. The value of the chunk_size expression must be the same for all the
threads in the team.

Section Construct
• The section constructs are included in the sections derivative.
• It is non iterative construct.
• It declares a set of constructs that are to be distributed among the
team members. Each section is executed once by a thread in the team.
• Each section preceded by a section directive and the section directive
is optional for the first section.
• The section directive must appear within the lexical extent of the
sections direfctive.
• Syntax
#pragma omp sections [clause [[,] clause]…] new line
{
[#pragma omp section new line]
structured block
[#pragma omp section new line]
 structured block
-----
-----
}
• Whole program is divided into sections; each section is run by single
thread. Each section runs independently of other sections. Multiple threads
concurrently handle the multiple sections sometimes.
• If threads are more than sections, then threads become idle, they have
no section to run.
• The clause may be one of the following:
private (variable list)
first private (variable list)
last private (variable list)
reduction (operator: variable list)
nowait
• Restriction to the section directive are:
1. A ‘section’ directive must appear within the lexical extent of the
sections directive.
2. Only a single ‘nowait’ clause can appear on a sections directive.
Single construct
• Single construct is hold by the single directive. It specifies that the
associated structured block is specified by only one thread in the team.
#pragma omp sections [clause [[,] clause]…] new line
structured block

• The clause may be one of the following:

private (variable list)
first private (variable list)
copy private (variable list)
nowait

• Restriction to the section directive are:

1. Only a single ‘nowait’ clause can appear on a single directive.
2. The copy private clause must not be used with the nowait caluse.

2.5. Combined Parallel Work Sharing Constructs

• These constructs are the combination or we say a kind of combination of

the parallel and work sharing construct.
• It is called ‘a kind of combination’ because it is not a total mixture of
parallel and work sharing constructs because only one work sharing
construct is used.

• The meaning of this directive is similar to those of both parallel and work
sharing construct.

1. Parallel for construct

2. Parallel sections construct

 The Parallel for construct is included in the parallel for directive. It is

a parallel region that contains only single ‘for’ directive.

 This directive allows all the clauses of parallel directive and for
directive except the nowait clause.

 Syntax:

#pragma omp parallel for [clause [[,] clause]…] new line

for loop

 The parallel sections construct is included in the parallel section

directive. It specifies the parallel region that contains only ‘sections’
directive. It can make use of any of the clauses used by parallel and
sections directive except the ‘nowait’ clause.

 Syntax:

#pragma omp parallel sections [clause [[,] clause]…] new line

[#pragma omp section new line]

structured block

[#pragma omp section new line]

structured block

-----

-----