VECTORS IN PETSc

CREATION AND DESTRUCTION

Vectors in PETSc are stored in data structure called Vec.

Vector declaration example

Vec x;

The structure must be declared and created using one of the creation routines before it is
used. Vector should be destroyed after it is no longer needed.

Creation routines do not set any values of vector's elements. This can be done later using
functions described in section 'Setting values'. Also, PETSc is responsible for allocating
memory
for storing values.

Creation

All creation, duplication and destruction routines are collective.

To create a sequential vector use

VecCreateSeq(MPI_Comm comm, int m, Vec *x);

This routine creates a sequential (residing on one process only) vector x of size m.
Communicator comm should be MPI_COMM_SELF.

Example (error checking omitted to increase readability)

#define X_SIZE 100

Vec x;
VecCreateSeq(MPI_COMM_SELF, X_SIZE, &x);

To create a distributed vector use

VecCreateMPI(MPI_Comm comm, int m, int M, Vec *x);

This routine creates a vector over an MPI communicator comm. The vector will be distributed
among processes belonging to comm. M is the vector size, and m is the number of vector
elements to be stored on current process (the one making the call). The programmer may
order PETSc to determine m or M, but not both (use PETSC_DECIDE as argument).
If local size is to be computed automatically, PETSc will distribute the vector with an equal
(or as close as possible) number of elements on each process. It can later be found using:

VecGetLocalSize(Vec v, int *size);

If global size was unspecified, its value may be retrieved using:

VecGetSize(Vec v, int *size);

In both cases size will contain the needed value. These two routines are not collective.

Example

#define X_SIZE 100

Vec x;

// We know the vector size, so we let PETSC compute

// local segment size
VecCreateMPI(PETSC_COMM_WORLD, PETSC_DECIDE, X_SIZE, &x);

// Get the computed local segment size

int x_loc_size;
VecGetLocalSize(x, &x_loc_size);

Creation in separate steps

Alternative method of creating a vector is using separate routines. Unlike VecCreateSeq or

VecCreateMPI where properties are set immediately, this way allows easier setting of
properties according to additional conditions. All routines in this paragraph are collective.

1. VecCreate(MPI_Comm comm, Vec *v);

2. VecSetType(Vec v, VecType type);

This sets the vector type, type equal to VECSEQ sets sequential type, VECMPI sets the
parallel type.

3. VecSetSizes(Vec v, int m, int M);

sets vector local and global size, PETSC_DECIDE can be used as in above routines. The
vector is now created and ready to use (set or copy values, assemble).

Example

#define X_SIZE 100

// Create vector as in the previous example

Vec x;

VecCreate(PETSC_COMM_WORLD, &x);
VecSetType(x, VECMPI);
VecSetSizes(x, PETSC_DECIDE, X_SIZE);
Creation by Copying

Another way of creating a vector is to use an existing one. To create a vector with parameters
based on another one use

VecDuplicate(Vec v, Vec *newvec);

This copies the structure and all attributes of v to the new vector newvec, but does not copy
the values.

Many vectors at once can be created basing on an existing one using

VecDuplicateVecs(Vec v, PetscInt m, Vec *newvecs[]);

This routine creates an array of m vectors which are copies of vector v. Again, values are not
copied.

Example

#define X_SIZE 100

#define Z_ARRAY_SIZE 10

Vec x;
VecCreateMPI(PETSC_COMM_WORLD, PETSC_DECIDE, X_SIZE, &x);

Vec y;
// y will have the same properties as x, values will not be copied
VecDuplicate(x, &y);

// Create an array of vectors

Vec z_array[Z_ARRAY_SIZE];

// Each vector in the array will have the same properties as x,

// values will not be copied
VecDuplicateVecs(x, Z_ARRAY_SIZE, &z_array);

Destruction

To destroy a vector use

VecDestroy(Vec v);

To destroy an array of vectors use

VecDestroyVecs(Vec vs[], PetscInt m);

This routine should be used to destroy an array of vectors previously created with
VecDuplicateVecs.

Example
// Assume the vectors were created as in previous example

VecDestroy(x);
VecDestroy(y);
VecDestroyVecs(z_array, Z_ARRAY_SIZE);

Setting values

Once the vector is created, values of its elements can be set.

The following routine assigns a given value to all elements of a vector:

VecSet(Vec x, PetscScalar value);

Another way of setting values is using the following function:

VecSetValues(Vec x, PetscInt ni, const PetscInt ix[], const

PetscScalar y[], InsertMode mode);

x - vector to be modified
y - array of values to be inserted
ix - destination indices of values in y
ni - number of values in y
mode - two possible values: INSERT_VALUES causes the values to be copied into the
vector, ADD_VALUES adds values to existing ones

After the VecSet or VecSetValues routines are called, the vector assembly phase should be
executed:

VecAssemblyBegin(Vec x);
VecAssemblyEnd(Vec x);

VecAssemblyEnd uses the following options:

-vec_view Prints the assembled vector in ASCII format (to stdout)
-vec_view_matlab Prints the assembled vector in ASCII Matlab format (to stdout)
-vec_view_matlab_file Prints the assembled vector in Matlab format (to
matlaboutput.mat)

The programmer may call other routines between these two calls. Any number of
VecSetValues calls can be executed on a vector using both modes of insertion, but before
different mode than in previous call is used, assembly phase must be executed.

Example

#define X_SIZE 100

Vec x;
VecCreateMPI(PETSC_COMM_WORLD, PETSC_DECIDE, X_SIZE, &x);

// Set all values to 1.0

VecSet(x, 1.0);

// Assume that we want to set first 10 values to doubled values

// of their indices

int ni = X_SIZE; // length of arrays

int i;

// Prepare indices array

int ix[X_SIZE];
for (i=0 ; i<ni ; i++)
ix[i] = i;

// Prepare values array

PetscScalar y[X_SIZE];
for (i=0 ; i<ni ; i++)
y[i] = i * 2;

// Execute setting the values

VecSetValues(x, ni, ix, y, INSERT_VALUES);

// Execute assembly phase

VecAssemblyBegin(x);
VecAssemblyEnd(x);

When setting a single value, use the following:

VecSetValue(Vec x, int row, PetscScalar value, InsertMode

mode);

This inserts value at position row into x. Depending on mode, value is inserted (mode is
INSERT_VALUES) or added to existing one (ADD_VALUES). Rules concerning assembly
phase are the same as in the case of VecSetValues.

Example

#define X_SIZE 100

Vec x;
VecCreateMPI(PETSC_COMM_WORLD, PETSC_DECIDE, X_SIZE, &x);

// We want the values to be equal their index, doubled

int i;
for (i=0 ; i<X_SIZE ; i++)
VecSetValue(x, i, i*2, INSERT_VALUES);

// Execute assembly phase

VecAssemblyBegin(x);
VecAssemblyEnd(x);

If it's necessary for a vector to have the same values as another one, the programmer should
use:

VecCopy(Vec x, Vec y);

This executes yi = xi. Types of vectors (sequential or parallel) don't have to match.
Accessing values

In order to read values of a vector v the programmer should use the following routine:

VecGetArray(Vec v, PetscScalar **array);

The argument array should be an address of a pointer, which will point to an array of
elements owned by the process. When possible, PETSc does not copy any values, just sets the
pointer to an existing array. After calling this routine, elements can be accessed like a
standard array. Note that the process can only access values stored locally.

Changing values in the array takes immediate effect - since this is the actual array used by
vector object to store values, they can be read/set directly.

When the array is no longer necessary, the programmer should call

VecRestoreArray(Vec v, PetscScalar **array);

to free any resources that were allocated. v is the vector of which elements are in the array,
and array is the pointer acquired with VecGetArray.

It is possible to acquire arrays of elements of vectors created with VecDuplicate :

VecGetArrays(const Vec x[], PetscInt n, PetscScalar **a[]);

x - the previously acquired array of vectors,

n - its length (number of vectors),
a - variable to contain the pointer.

Arrays obtained this way should be freed:

VecRestoreArrays(const Vec x[], PetscInt n, PetscScalar **a[]);

x - the array of vectors elements of which are in the array,

n - the array length,
a - pointer acquired with VecGetArrays.

If the vector has been created so that PETSc automatically assigns elements to processes, it
may be necessary to know the range of elements stored locally. The following can be used:

VecGetOwnershipRange(Vec vec, int *low, int *high);

vec - the vector,

low - will be set to the lowest index of locally owned elements of vec,
high - will be set to the highest index of locally owned elements of vec, increased by 1.

Example

#define X_SIZE 100

Vec x;
VecCreateMPI(PETSC_COMM_WORLD, PETSC_DECIDE, X_SIZE, &x);

// Get local array of x and its size

PetscScalar *xa;
VecGetArray(x, &xa);

int x_loc_size;
VecGetLocalSize(x, &x_loc_size);

// Again, we want the values to be equal their doubled index,

// so we need to know the global indices of the elements.
// Thus we need to acquire the local range.
int x_low, x_high;
VecGetOwnershipRange(x, &x_low, &x_high);

// Set the values directly in the array

int i;
for (i=0 ; i<x_loc_size ; i++)
xa[i] = (i+x_low)*2;

// Restore the array

// Note that there is no assembly phase
VecRestoreArray(x, &xa);

VECTOR OPERATIONS
Vectors in the following routines do not have to have matching types.

Norm

VecNorm(Vec x, NormType type, double *r);

This routine computes the norm of x, where type is one of the following:

NORM_1 r = Σi |xi|
NORM_2 r = (Σixi2)1/2
NORM_INFINITY r = maxi |xi|

Dot product

VecDot(Vec x, Vec y, PetscScalar *r);

r = xHy, xH is the conjugate transpose of x

VecTDot(Vec x, Vec y, PetscScalar *r);

r = xTy

It's possible to compute the dot product of many pairs of vectors at once:

VecMDot(Vec x, PetscInt nv, const Vec y[], PetscScalar *val);

VecMTDot(Vec x, PetscInt nv, const Vec y[], PetscScalar *val);
Both compute the same values as previous routines, but for all pairs (x, y[i]).

y - the array of vectors y,

nv - length of y,
val - pointer to an array of variables which will store the respective values.

Multiplication

Key to the names: AX- multiplication (a*x, a is a scalar) XPY - x+y, y is usually the result,
for example: VecAXPY is "y = ax + y".

VecAXPY(Vec y, PetscScalar a, Vec x) y = ax + y

VecAYPX(Vec y, PetscScalar a, Vec x) y = ay + x
VecWAXPY(Vec w, PetscScalar a, Vec x, Vec y) w = ax + y
VecAXPBY(Vec y, PetscScalar a, PetscScalar b, Vec x) y = ax + by

Other

VecShift(Vec x, PetscScalar s) xi = s + xi (each element)

VecScale(Vec x, PetscScalar a) x = ax

VecReciprocal(Vec x) xi = 1/xi (each element)

VecSwap(Vec x, Vec y) swaps x and y (x = y, y = x)

VecEqual(Vec vec1, Vec vec2, PetscTruth *flg)

flg will be set to PETSC_TRUE if vectors are equal, PETSC_FALSE otherwise

VECTOR PRINTING

Vector contents can be printed/written manually (after gaining access to the array), but it's
often useful to use PETSc built-in printing routines, for example

VecView(Vec vec, PetscViewer viewer);

This routine writes vector vec contents where the viewer object defines.

This routine is collective.

The simplest method is to use predefined viewer, PETSC_VIEWER_STDOUT_WORLD:

Vec x;
/* Create vector and use it */
VecView(x, PETSC_VIEWER_STDOUT_WORLD);

In this case, the vector is sent to the first process (if necessary) and printed onto the standard
output. Values of elements are printed; also processes that own the vector segment (element
group) are printed at the beginning of each segment.
Another way is to create an ASCII viewer associated with a file. Writing to it will result in
information written to the designated file. Use the following:

PetscViewerASCIIOpen(MPI_Comm comm, const char name[],

PetscViewer *viewer);

This routine is collective. The argument comm is the MPI communicator containing
processes which will be using the viewer, name is the name of the file to which information
will be printed and viewer is the viewer object we want to create to be used later.

Example

Viewer viewer;
PetscViewerASCIIOpen(PETSC_COMM_WORLD, "outfile.txt", &viewer);

Vec x;

... create the vector and set values

// Print the vector to "outfile.txt" using the viewer

VecView(x, viewer);

The way which the vector is printed to the file can be decided by setting the viewer format:

PetscViewerSetFormat(PetscViewer viewer, PetscViewerFormat

format);

This routine is collective.

This routine sets the format for printing operations in the given viewer. Some of the possible
format values are:

PETSC_VIEWER_ASCII_DEFAULT - default format – prints element values and

process owning the elements,
PETSC_VIEWER_ASCII_INDEX
PETSC_VIEWER_ASCII_SYMMODU
just the values,
PETSC_VIEWER_ASCII_MATLAB
- additionally prints indices of elements,
- no indices and no process identifiers are printed,
- Matlab-readable format.

There are more formats, but their output does not differ from those shown in the case of
vectors.

Examples

The following are effects of printing a sample 10-element vector distributed among 3
processes.

Printing to standard output using VecView(x, PETSC_VIEWER_STDOUT_WORLD):

Process [0]
0
2
4
6
Process [1]
8
10
12
Process [2]
14
16
18

Printing to file using PETSC_VIEWER_ASCII_DEFAULT format:

Process [0]
0
2
4
6
Process [1]
8
10
12
Process [2]
14
16
18

Printing to file using PETSC_VIEWER_ASCII_INDEX format:

Process [0]
0: 0
1: 2
2: 4
3: 6
Process [1]
4: 8
5: 10
6: 12
Process [2]
7: 14
8: 16
9: 18

Printing to file using PETSC_VIEWER_ASCII_SYMMODU format:

0.0000000000000000e+00
2.0000000000000000e+00
4.0000000000000000e+00
6.0000000000000000e+00
8.0000000000000000e+00
1.0000000000000000e+01
1.2000000000000000e+01
1.4000000000000000e+01
1.6000000000000000e+01
1.8000000000000000e+01
Printing to file using PETSC_VIEWER_ASCII_MATLAB format:

Vec_0 = [
0.0000000000000000e+00
2.0000000000000000e+00
4.0000000000000000e+00
6.0000000000000000e+00
8.0000000000000000e+00
1.0000000000000000e+01
1.2000000000000000e+01
1.4000000000000000e+01
1.6000000000000000e+01
1.8000000000000000e+01
];

Any text can be printed to the viewer, just like printf writes to the standard output, using:

PetscViewerASCIIPrintf(PetscViewer viewer, const char format[],

...);

This routine is not collective.

The first argument is the viewer, and others are as in the case of printf. Only the first process
in the communicator can print this way, calls on others have no effect.

When the viewer is no longer needed, it can be disposed of using:

PetscViewerDestroy(PetscViewer viewer);

This routine is collective.

Distributed vector example

Suppose we have a file vec1.txt containing a vector:

10
0.5
1
1.5
2
2.5
3
3.5
4
4.5
5

The first value is the number of elements and others are element values. Values appear in the
same order they are in the vector. Values are in separate lines (the newline '\n' is the
separator).
We will load the vector on one process (the file resides on one machine only), distribute it
among processes and print the content (and some other information) on each process. Number
of processes and length of the vector are assumed not to be known (length must be read from
file at runtime).

Note that we first read the vector length and distribute it using MPI_Bcast, and then create
the vector, as we want PETSc to determine the local size.

We create the vector using VecCreateMPI. Another way – separate steps – is shown in the
code, commented out, which allows the user to set the vector type using database options..

PetscSynchronizedPrintf is used to print information to the standard output. This function

prints on one process only, and makes sure that the output from one process is not interrupted
by another one. Also, (series of) the messages appear in increasing order with respect to
process rank (rank 0, rank 1, ...). Arguments to PetscSynchronizedPrintf are the same as to
printf in the standard library. PetscSynchronizedFlush call is required to actually print the
messages.

Additionally, VecView was used to show how the vector is displayed by the standard PETSc
function. The vector is sent to one process and only that process writes the results (when using
PETSC_VIEWER_STDOUT_WORLD as second argument).

/* Text to be displayed when '--help' parameter is used */

static char help[] = "Distributed vector example.\n\n";

#include "petscvec.h"

#include <stdio.h>
#include <stdlib.h>

#undef __FUNCT__
#define __FUNCT__ "main"
int main(int argc, char **args)
{
/* Number of processes and current process rank */
int size, rank;

/* Vector declaration */
Vec v;

PetscInt v_size, /* vector length */

v_loc_size, /* number of vector elements locally */
v_loc_start, /* index of first element on process */
v_loc_end; /* index of last element + 1 */

/* Variable to store error code */

PetscErrorCode ierr;

/* Name of file containing the vector */

char vecfile[] = "vec1.txt";

/* Value read from vector file */

double val;

/* Initialization - MPI_Init and others*/

PetscInitialize(&argc, &args, (char *) 0, help);

/* Acquire number of processes and rank of current process */

MPI_Comm_size(PETSC_COMM_WORLD, &size);
MPI_Comm_rank(PETSC_COMM_WORLD, &rank);
/* Vector file */
FILE *vecf;

int i;

/* Read vector size from file (only process 0 - root) */

if (rank==0)
{
vecf = fopen(vecfile, "r");

fscanf(vecf, "%d\n", &v_size);

}

/*
* Process 0 sends vector size to all other processes
* and the rest receives it.
* We assume that PetscInt is the standard int (default behaviour),
* so we can cast safely.
*/
int send_vsize = (int) v_size;
MPI_Bcast(&send_vsize, 1, MPI_INT, 0, PETSC_COMM_WORLD);
v_size = (PetscInt) send_vsize;

/*
* Vector creation
* We decide the vector type (parallel) immediately
*/

ierr = VecCreateMPI(PETSC_COMM_WORLD, PETSC_DECIDE, v_size, &v); CHKERRQ(ierr);

/*
* Vector creation, separate steps
* We decide about vector attributes in separate steps.
*/

/*
ierr = VecCreate(PETSC_COMM_WORLD, &v); CHKERRQ(ierr);
ierr = VecSetType(v, VECMPI); CHKERRQ(ierr);
ierr = VecSetSizes(v, PETSC_DECIDE, v_size); CHKERRQ(ierr);
*/

/*
* Root process reads element values from file and sets them,
* other processes do nothing - they will receive their values
* automatically later.
*/
if (rank==0)
{
for (i = 0 ; i<v_size ; i++)
{
fscanf(vecf, "%lg\n", &val);
ierr = VecSetValue(v, i, val, INSERT_VALUES); CHKERRQ(ierr);
}
}

/* Assembly phase - send values to processes that require them. */

ierr = VecAssemblyBegin(v); CHKERRQ(ierr);
ierr = VecAssemblyEnd(v); CHKERRQ(ierr);

/* Read the size of locally owned vector part, and its range */
ierr = VecGetOwnershipRange(v, &v_loc_start, &v_loc_end); CHKERRQ(ierr);
ierr = VecGetLocalSize(v, &v_loc_size); CHKERRQ(ierr);

/* Gain pointer to an array containing locally owned values */

PetscScalar *v_loc_array;
 ierr = VecGetArray(v, &v_loc_array); CHKERRQ(ierr);

/*
* Print information about vector to standard output,
* each process reports its own data. Messages from different processes
* do not get mixed.
*/
PetscSynchronizedPrintf(PETSC_COMM_WORLD, "Process %d:\n", rank);
PetscSynchronizedPrintf(PETSC_COMM_WORLD, "v size = %d\n", v_size);
PetscSynchronizedPrintf(PETSC_COMM_WORLD, "v local size = %d\n", v_loc_size);
PetscSynchronizedPrintf(PETSC_COMM_WORLD, "v local start of range = %d\n",
v_loc_start);
PetscSynchronizedPrintf(PETSC_COMM_WORLD, "v local end of range = %d\n",
v_loc_end);

for (i = 0 ; i<v_loc_size ; i++)

PetscSynchronizedPrintf(PETSC_COMM_WORLD, "v[%d] = %lg\n", i+v_loc_start,
v_loc_array[i]);

PetscSynchronizedPrintf(PETSC_COMM_WORLD, "\n");

PetscSynchronizedFlush(PETSC_COMM_WORLD);

/* Use built-in routine to print vector content */

ierr = VecView(v, PETSC_VIEWER_STDOUT_WORLD); CHKERRQ(ierr);

ierr = VecRestoreArray(v, &v_loc_array); CHKERRQ(ierr);

/* Destroy the vector object */

ierr = VecDestroy(v); CHKERRQ(ierr);

/* Close vector file */

if (rank==0)
{
fclose(vecf);
}

/* Finalization - MPI_Finalize, print additional information if requested */

ierr = PetscFinalize(); CHKERRQ(ierr);

return 0;
}

The sample output from the program is shown below. The program was run on 3 processes.

Process 0:
v size = 10
v local size = 4
v local start of range = 0
v local end of range = 4
v[0] = 0.5
v[1] = 1
v[2] = 1.5 highest index + 1
v[3] = 2

Process 1:
v size = 10
v local size = 3
v local start of range = 4
v local end of range = 7 highest index
v[4] = 2.5
v[5] = 3
v[6] = 3.5
Process 2:
v size = 10
v local size = 3
v local start of range = 7
v local end of range = 10
v[7] = 4
v[8] = 4.5
v[9] = 5

Process [0]
0.5
1
1.5
2
Process [1]
2.5 VecView(v, PETSC_VIEWER_STDOUT_WORLD) output
3
3.5
Process [2]
4
4.5
5