Author's Accepted Manuscript

ModEM: A Modular System for Inversion of

Electromagnetic Geophysical Data
Anna Kelbert, Naser Meqbel, Gary D. Egbert,
Kush Tandon
PII: S0098-3004(14)00021-1
DOI: http://dx.doi.org/10.1016/j.cageo.2014.01.010
Reference: CAGEO3327
To appear in: Computers & Geosciences
Received date: 11 June 2013
Revised date: 15 November 2013
Accepted date: 26 January 2014
Cite this article as: Anna Kelbert, Naser Meqbel, Gary D. Egbert, Kush Tandon,
ModEM: A Modular System for Inversion of Electromagnetic Geophysical
Data, Computers & Geosciences, http://dx.doi.org/10.1016/j.cageo.2014.01.010
This is a PDF file of an unedited manuscript that has been accepted for
publication. As a service to our customers we are providing this early version of
the manuscript. The manuscript will undergo copyediting, typesetting, and
review of the resulting galley proof before it is published in its final citable form.
Please note that during the production process errors may be discovered which
could affect the content, and all legal disclaimers that apply to the journal
pertain.
www.elsevier.com/locate/cageo
ModEM: A Modular System for Inversion of 1
Electromagnetic Geophysical Data 2
Anna Kelbert
a
, Naser Meqbel
b,a
, Gary D. Egbert
a
, Kush Tandon
c,a
3
a
College of Earth, Ocean and Atmospheric Sciences, Oregon State University, 104 4
CEOAS Admin Bldg., Corvallis, OR 97331-5503, USA 5
b
Helmholtz Centre Potsdam, GFZ German Research Centre for Geosciences, Potsdam, 6
Germany 7
c
Bluware Inc., Houston, TX 77063, USA 8
Abstract 9
We describe implementation of a modular system of computer codes for inver-
sion of electromagnetic geophysical data, referred to as ModEM. The system
is constructed with a ne level of modular granularity, with basic compo-
nents of the inversion forward modeling, sensitivity computations, inversion
search algorithms, model parametrization and regularization, data function-
als interchangeable, reusable and readily extensible. Modular sensitivity
computations and generic interfaces to parallelized inversion algorithms pro-
vide a ready framework for rapid implementation of new applications or inver-
sion algorithms. We illustrate the codes versatility and capabilities for code
reuse through implementation of 3D magnetotelluric (MT) and controlled-
source EM (CSEM) inversions, using essentially the same components.
Keywords: Geophysics, numerical modeling, inversion, magnetotellurics, 10
controlled-source electromagnetics, sensitivities, parallelization, code reuse, 11
object-oriented programming. 12
Preprint submitted to Computers & Geosciences January 29, 2014
1. Introduction 13
Egbert and Kelbert (2012) (hereinafter EK12) present a general discrete 14
formulation for linearized electromagnetic (EM) inverse problems in the fre- 15
quency domain, treating a range of EM geophysical techniques and inversion 16
approaches in a unied notational framework. This framework highlights the 17
elements, and dependencies between elements, that are common across ap- 18
plications and inversion algorithms, providing a basis for a general modular 19
system of computer codes for EM geophysical inverse problems. EK12 pro- 20
vided a brief overview of such a system, which they referred to as ModEM: 21
the Modular system for Electromagnetic inversion. Here we present a more 22
detailed description of this software package, implemented in the Fortran 95 23
programming language. 24
25
Geophysical electromagnetic inversion approaches and parallel computer 26
codes have developed rapidly in the past decade or so (notably, Newman 27
and Alumbaugh (2000); Zhdanov and Hursan (2000); Haber et al. (2004); 28
Newman and Boggs (2004); Kelbert et al. (2008); Siripunvaraporn and Eg- 29
bert (2009); Avdeev and Avdeeva (2009) and others; see also review papers 30
Avdeev (2005) and Siripunvaraporn (2011) and the overview in EK12). While 31
most of these eorts focused on a specic EM inverse problem, solved with a 32
particular computational approach, recent eorts in the context of joint in- 33
version (e.g., Moorkamp et al. (2011); Commer and Newman (2009); Stefano 34
et al. (2011)), have seen the development of more exible inversion frame- 35
works, which allow a single inversion algorithm to be applied to a range of 36
dierent data types. Our focus here is on a ner granularity of modulariza- 37
2
tion, with the goal of making the basic components of the inversion for- 38
ward modeling, sensitivity computation, inversion search algorithms, model 39
parametrization and regularization, and data functionals interchangeable, 40
reusable and readily extensible. Thus, while ModEM is exible enough to be 41
applied to a range of EM data types, and indeed to joint inversion, it also 42
provides a code base suitable for rapid development and prototyping of new 43
parallel inversion algorithms, and for experimentation with dierent model 44
parametrization and regularization schemes. To the extent possible (given 45
limitations of Fortran 95; Akin (2003)) we have taken an object oriented 46
approach to maximize code reuse, and to provide templates for rapid devel- 47
opment of new applications. We also use the terminology of this approach 48
in our discussion here. 49
50
To allow readers to more easily follow the general development, we provide 51
a brief overview of the theory and notation of EK12 in Section 2. However, 52
as our focus is on program structure and logic, mathematical formulae re- 53
quired for implementation of specic routines are not repeated here; readers 54
are referred to appropriate sections in EK12 for such details. ModEM can be 55
roughly organized into three functional levels which we discuss in successive 56
sections. Section 3 provides an overview of ModEM, and describes imple- 57
mentation of top level generic inversion and sensitivity routines, including 58
a coarse grained general parallelization. These top-level routines, and the 59
parallelization (discussed in Section 4) are intended to be directly reusable 60
for a wide range of dierent problems. This generality requires that all basic 61
objects manipulated by the inversion (forward operators, data functionals, 62
3
model parameters, electric and/or magnetic eld solution vectors, etc.) fol- 63
low uniform conventions. A distinctive characteristic of our formulation is 64
the capability to accommodate most modications in the geophysical forward 65
problem (details of the solver, types of observable, etc) and easily translate 66
these into implementation of the needed numerical derivative. This unifor- 67
mity is provided through the interface level, discussed in Section 5. In 68
eect this level of ModEM provides specic instances of the abstract classes 69
used by top-level routines. Actual computations for specic problems (e.g., 70
forward problem solutions, evaluation of EM eld components at specied 71
data site locations) are implemented in the numerical discretization mod- 72
ules, described in Section 6. Section 7 provides specic examples for two 73
3D EM inverse problems: magnetotelluric (MT) and controlled-source elec- 74
tromagnetic (CSEM). The two methods, which dier with regard to sources 75
and data types, are implemented through variations of the interface layer. 76
This example illustrates that due to the ne grained modularity of our ap- 77
proach, not only the parallel inversion algorithms, but also the numerical 78
discretization (including the nite-dierence 3D EM forward solver) and, 79
more importantly, the sensitivity computations may be eectively reused for 80
these two very dierent EM techniques. 81
82
2. Preamble: Theory and Notation 83
The theory of geophysical inversion has been previously described in 84
generic terms in Parker (1994); Zhdanov (2002); Tarantola (2005), among 85
others, and the general principles of discrete sensitivity computations for non- 86
4
linear electromagnetic inverse problems in geophysics discussed, for example, 87
in McGillivray et al. (1994), Newman and Hoversten (2000) and EK12. 88
In our discussion of these ideas throughout this paper, we use the notation 89
of EK12, which is outlined in Table 1. For completeness we summarize key 90
points here. ModEM provides a general framework for solving regularized 91
EM inverse problems, i.e., minimization of a penalty functional of the form 92
(m, d) = (d f (m))
T
C
1
d
(d f (m)) + (mm
0
)
T
C
1
m
(mm
0
) (1)
to recover an Earth conductivity model parameter vector m, which provides 93
an adequate t to a data vector d. In (1), C
d
is the covariance of data 94
errors, f (m) denes the forward mapping, m
0
is a prior or rst guess model 95
parameter, is a trade-o parameter, and C
m
(or more properly
1
C
m
) 96
denes the model covariance or regularization term. 97
The forward mapping requires solution of the frequency domain EM par- 98
tial dierential equation (PDE), which in discrete form is written generically 99
as 100
S
m
e = b. (2)
The subscript m here denotes the dependence of the PDE operator on a 101
specic model parameter, and in the following is often omitted; e represents 102
the discrete EM eld solution; and b is the forcing (boundary conditions 103
and/or source terms). Typically e will represent only the primary eld (e.g., 104
electric) that is actually solved for; the other dual eld (e.g., magnetic) is 105
then computed via a simple transformation operator h = Te. Simulated 106
5
observations are computed from the solution e (and possibly m) via 107
d
j
= f
j
(m) =
j
(e(m), m). (3)
Using the chain rule, a general expression for the Jacobian (or sensitivity 108
matrix) J (= f /m) can be given, in the vector notation, as 109
f
j
m

m=m
0
=
_

j
e

e=e
0
,m=m
0
_
e
m

m=m
0
+

j
m

m=m
0
(4)
where e
0
is the solution to (2) for model parameter m
0
. A simple calculation 110
shows 111
e
m

m=m
0
= S
1
m
0
_

m
(S
m
e
0
)
_
m=m
0
= S
1
m
0
P. (5)
So, in matrix notation the Jacobian can be expressed as 112
J = LS
1
m
0
P+Q. (6)
The matrix P depends on details of the numerical model implementation and 113
the conductivity parametrization. An explicit formula for P can be given, 114
assuming the forward operator can be written 115
S
m
e S
0
e +U ((m) Ve), (7)
where S
0
, U and V are some linear operators that do not depend on the 116
model parameter vector m, (m) is a (possibly non-linear) mapping from 117
the model parameter space to the numerical grid, and ( ) denotes the 118
6
component-wise multiplication of vectors. Then we have 119
P = Udiag(Ve
0
)
m
0
, (8)
where
m
0
is the Jacobian of the model parameter mapping (m) evaluated 120
at the background model parameter m
0
. All of the usual forms for 2D and 121
3D EM induction operators can be expressed as in (7); e.g., for the second- 122
order 3D staggered-grid nite dierence equation for the electric eld (with 123
possible source term j
s
) 124
E + iE = j
s
(9)
S
0
corresponds to the discrete curl-curl operator, U iI, V I, and 125
(m) (m) is a mapping from the model parameter space to the cell 126
edges, where electric eld components are dened. 127
The matrix L in (6) represents the linearized data functionals. This can be 128
decomposed into two sparse matrices as L = A
T

T
, where columns of are 129
sparse vectors which represent evaluation functionals for point observations 130
of the electric and magnetic elds. The operator T is generally involved in 131
evaluation of dual eld components. The matrix A depends on details of 132
the (generally non-linear) observation functionals (e.g., impedance, apparent 133
resistivity), which may combine magnetic and electric measurements from 134
one or more locations. More explicitly, each row of L takes the form 135
(l
j
)
T
=
_

j
e

e
0
,m
0
_
T
=
K
P

k=1
a
P
jk

P
k
+
K
D

k=1
a
D
jk
_
T
T

D
k

(10)
7
where
P
k
and
D
k
are sparse vectors dened on the primary and dual grids 136
(see EK12, Sec. 5.2.3), and l
j
is a primary grid sparse vector. 137
When either the evaluation functionals, or the eld transformation op- 138
erator T have an explicit dependence on the model parameter there is an 139
additional term in the sensitivity matrix, which we have denoted Q in (6). 140
Rows of this matrix can be eectively computed using the expression 141
(q
j
)
T
=
_

j
m

e
0
,m
0
_
T
=
T
m
0
_
K
D

k=1
a
D
jk

T
T
(m
0
),e
0

D
k
_
, (11)
following the notation of Table 1. Note that the expression in the square 142
bracket in (11) is also a sparse vector on the primary or dual grid, depending 143
on where the electrical conductivity (or resistivity) (m) is dened. 144
The Jacobian represents a linear mapping, giving the perturbation to the 145
data resulting from a model parameter perturbation (d = Jm). A wide 146
range of gradient-based inversion algorithms make use of this operator, along 147
with the transpose or adjoint (m = J
T
d). ModEM does not necessarily 148
(or even typically) form the Jacobian, or the component matrices in (6), 149
but rather implements the solver for the discrete system S
1
m
, the operators 150
P, L, Q, and the compound Jacobian operator J, together with adjoints. 151
These operators, together with model and data covariances, were then used 152
to implement a range of gradient-based inversion algorithms. As discussed in 153
EK12, and in detail below, in most EM inverse problems there is signicant 154
structure to the data vector, implied by the multiplicity of transmitters and 155
receivers. This structure is also reected in the Jacobian and the component 156
matrices, and thus in the organization of ModEM. 157
8
Symbol Represents
m, m
0
M model parameter vectors
(m) : M S
P,D
mapping from model parameter to primary or dual grid

m
0
: M S
P,D
/m, evaluated at m
0
C
m
: M M model covariance
e, e
0
S
P
solution vectors on the primary grid
d D data vector

j
(e(m), m) : S
P
C jth data functional, in general non-linear
l
j
S
P

j
/e, evaluated at e
0
, m
0
; jth row of L
q
j
M
j
/m, evaluated at e
0
, m
0
; jth row of Q
L : S
P
D sparse matrix constructed from l
j
Q : M D sparse matrix constructed from q
j
S
1
m
: S
P
S
P
forward solver; S
m
e = b
P : M S
P
operator (S
m
e
0
) /m, evaluated at m
0
J : M D full Jacobian /m; J = LS
1
m
0
P+Q
T : S
P
S
D
linear mapping from the primary to dual grid

T
(m
0
),e
0
: S
P,D
S
D

_
T
(m)
e
0
_
/, evaluated at (m
0
)

P
S
P
primary grid interpolation coecients (sparse)

D
S
D
dual grid interpolation coecients (sparse)
Table 1: List of notation from Egbert and Kelbert (2012). Here, M indicates the model
space, D the data space, and S
P,D
stand for the spaces of EM elds dened on primary
and dual grids, as dened in Egbert and Kelbert (2012).
9
3. Overview of the ModEM Program 158
The organization of ModEM is summarized in Figure 1, where we distin- 159
guish three general levels. At the top are generic components which imple- 160
ment parallelized inversion algorithms and sensitivity computations which 161
can be applied to a wide range of EM inverse problems. At the lowest level 162
are components which dene the basic discretization and numerical solution 163
approach used for the forward problem. These routines might be used for a 164
variety of frequency domain EM problems, with varying source and receiver 165
congurations. The middle layer provides an interface which hides specic de- 166
tails of the numerical implementation from the generic inversion modules, and 167
denes problem specic details (e.g., source and receiver congurations). In 168
this section, we guide the reader through the overall organization of ModEM 169
code, and discuss the top level, which is the core of ModEM. The interface 170
layer will be further discussed in Section 5, and the numerical discretization 171
layer in Section 6. 172
As summarized in Section 2, the basic data objects which are manipu- 173
lated in any inversion scheme include model (m) and data (d) vectors, and 174
EM solution and source elds (e and b). Except for the data vector these are 175
treated in all top level routines as essentially abstract classes, with no spe- 176
cic implementation details referenced by the generic inversion or sensitivity 177
routines. These are implemented in Fortran 95 as data structures with stan- 178
dardized type names and interfaces, allowing implementations for dierent 179
problems to be used interchangeably within the inversion system. For each 180
of these data objects a standard series of methods must be dened (creation, 181
destruction, vector space methods, dot products, etc.), again with standard- 182
10
Model Space
m

Data Space
d

J
L,Q

,T

Mappings from Model
Space to Numerical Grid
,


Numerical
Forward Solver
Forward
Solver
(Driver)
Jacobian
Computations
Inversion Algorithms
(NLCG, DCG etc)
Numerical Grid
Interpolation on the
Numerical Grid
Solution
Space
Solver
Sensitivities
EM fields
Data Functionals
Data
Sensitivities
e,b

S
-1 P

DT RX
TX DT
TX TX
TX
Generic
Inversion
Interface
Layer
Numerical
Discretization
,l
j
,q
j

Workers
MPI
Master
Model parameter vector
space; regularization
Set sources, BCs,
efficiently manage
multiple TX
Model Map.:
EM field Interp.:
Data vector space,
organized by TX,RX
Figure 1: Schematic overview of ModEM. Conceptual modules, which are described in
Sections 36, are denoted by boxes, with dependencies indicated by arrows. Some boxes
are also marked with symbols to indicate the vectors and operators from Egbert and
Kelbert (2012) which they implement (see also Table 1). The shaded small boxes indicate
which dictionaries are used in each module. The (optional) Message Passing Interface
(MPI) module used for the parallel implementation is shown in lighter colors. Modules
marked MPI also require additional subroutines to allow transmission of derived data
types dened within these modules between processors.
11
ized interfaces. The model parameter m interacts more directly with the 183
numerical discretization level, and will be discussed further in Section 6. EM 184
solution (e) and source eld (b) objects are application specic, and actual 185
implementations of these abstract classes are dened in the interface layer, 186
discussed in Section 5. Note that we also make extensive use of a sparse vec- 187
tor representation for elements of the EM solution space, to allow ecient 188
representation of observation functionals. 189
In contrast to the other basic data objects, data vectors d have been im- 190
plemented with a xed structure which is accessible to, and heavily used by, 191
the top level inversion and sensitivity routines. We do this to allow ecient 192
treatment of the multi-transmitter/multi-receiver data sets that are common 193
to EM methods. For example, for 3D MT there are multiple frequencies, 194
each requiring separate forward solutions. For each frequency there will be 195
multiple sites, and at each site 8 real components in the impedance tensor, 196
which require solutions for two independent source polarizations for their 197
evaluation. Controlled source problems have a multiplicity of sources and 198
receivers, which may interact in dierent ways. Data vectors are thus or- 199
ganized according to three attributes which we refer to as transmitter, data 200
type, and receiver. The transmitter attribute uniquely denes the forward 201
problem that must be solved, including both the specic partial dierential 202
equation (in general this will depend on frequency) as well as the sources and 203
boundary conditions. The receiver attribute is used to dene, in conjunction 204
with data type, the measurement process that must be applied to the forward 205
solution to allow comparison between measured and predicted data. These 206
attributes are used to dene a natural organization of the full data vector 207
12
d, which at the coarsest level consists of an array of structures correspond- 208
ing to dierent transmitters TX. Each of these in turn contains one or more 209
sub-structures for dierent data types DT, which store all components for 210
all receivers for that TX/DT pair. 211
The three data attributes TX, DT, RX are treated abstractly at the level 212
of the inversion and sensitivity modules. This is achieved by storing the 213
actual information associated with these attributes as lists, which we call 214
dictionaries. Thus, the transmitter (TX) dictionary has an entry for each 215
unique forward problem, providing any data such as the frequency or geom- 216
etry of the source required to set up and solve the forward problem. Entries 217
in the data type (DT) dictionary dene data functional types included in 218
the inversion, such as impedance, vertical eld transfer function, phase ten- 219
sor, apparent resistivity. The receiver (RX) dictionary provides information 220
about site locations, and if appropriate, orientation/conguration of the ob- 221
serving system. The dictionaries are only employed by the interface level 222
modules (their use is denoted by small corner boxes in Figure 1). This sep- 223
aration of the data values from their accompanying meta data information 224
makes the data vector structure completely generic, allows mixing data of 225
dierent types, and greatly simplies addition of new data types. At the 226
same time, tagging components of d with these generic attributes provides 227
enough information about the transmitter/receiver structure so that forward 228
modeling and sensitivity computations can be organized eciently. For ex- 229
ample, the transmitter attribute can be used to ensure that each required 230
forward problem is solved once (and only once), and then used to compute 231
predicted data (or implement appropriate sensitivity calculations) for all nec- 232
13
essary receivers and data types. 233
A key feature of ModEM is the ability to easily implement any linearized 234
inversion scheme that can be expressed in terms of the basic data objects d, 235
m, e and b, together with the forward mapping f (m), Jacobian J, and data 236
and model covariances C
m
and C
d
. From the perspective of the inversion 237
algorithms only a small number of conceptually simple operations need to be 238
implemented for the covariance operators: multiplication of model parame- 239
ter objects by C
m
, C
1
2
m
, and perhaps C

1
2
m
, and multiplication of data vector 240
objects by C
1
2
d
and C

1
2
d
. Interfaces for these symmetric covariance operators 241
are simple, with both inputs and outputs being model parameters or data 242
vectors, as appropriate. So far we have only implemented simple diagonal 243
data error covariances within the data vector class. Model covariances, which 244
are generally more complicated, are discussed further below in conjunction 245
with the model parameter class. General implementations of high level rou- 246
tines for forward and Jacobian operators are implemented in the sensitivity 247
module, following conventions given in Table 2. 248
Using these components we have so far implemented several inversion 249
algorithms including non-linear conjugate gradients (NLCG; e.g., Rodi and 250
Mackie (2001)), data space conjugate gradients (DCG; Siripunvaraporn and 251
Egbert (2007)), and the multi-transmitter hybrid CG-Occam scheme of Eg- 252
bert (2012). Example pseudo-code and further details are given in Appendix 253
A. 254
Jacobian routines are further modularized, using the general decomposi- 255
tion of (6) (or its transpose) to implement multiplication by J (or J
T
), using 256
the solver for the discrete system S
1
m
, and the operators P, L, Q (or their ad- 257
14
joints). For eciency, and to simplify parallelization, these computations are 258
organized by transmitter. Thus, generic routines in the sensitivity module 259
(see Table 2) implement multiplication by L and Q for a single transmitter. 260
Each row of these operators (l
j
and q
j
in Table 1) corresponds to a single 261
data type/receiver and denes the derivative of a single (generally non-linear 262
and multi-component) data functional
j
with respect to the EM solution e 263
(l
j
) and, if necessary, the model parameter m (q
j
). Individual rows, which 264
can generally (but not always) be represented by sparse vectors in the ap- 265
propriate space, are application dependent and are constructed by routines 266
in the interface layer. L and Q (or the corresponding adjoints) are called, 267
along with the solver and the operator P (or P
T
; see Section 5) to complete 268
Jacobian calculations for a single transmitter. Higher level routines (Jmult, 269
JmultT in Table 2) then loop over transmitters to complete the multiplication 270
by the full Jacobian. 271
The sensitivity module also provides routines (fwdPred) that implement 272
the full forward problem d = f (m), and for calculation of the full Jacobian 273
(calcJ). Note that fwdPred optionally returns the array of EM solutions 274
objects e computed for all unique transmitters, so that these can be saved 275
(e.g., after evaluation of data mist) and used for subsequent sensitivity 276
calculations (e.g., to evaluate the gradient of the penalty functional). 277
The organization of sensitivity computations we have described so far is 278
simple and general, but may not be ecient for all problems. As discussed 279
in EK12, in some cases computations with the Jacobian can be factored 280
for eciency into components that depend on the receiver and on the trans- 281
mitter. The simplest example is the controlled source cross-well imaging 282
15
problem considered by Newman and Alumbaugh (1997), with N
T
transmit- 283
ters and N
R
receivers. The full Jacobian for this problem can be constructed 284
from forward solutions for each transmitter, and adjoint solutions for each 285
receiver. Thus if N
T
N
R
the most ecient way to implement a Gauss- 286
Newton scheme is to pre-compute and save these N
T
+N
R
solutions and use 287
these to implement multiplication by J and J
T
(Newman and Alumbaugh, 288
1997). Implementation of such a scheme in ModEM is also straightforward 289
the basic implementation of routines Jmult and JmultT described above are 290
simply replaced, using an implementation that pre-computes and saves the 291
appropriate forward and adjoint solutions. All of the computations in these 292
variants are readily implemented using virtually the same components re- 293
quired for the basic versions outlined above. 294
In summary, routines in the sensitivity module manage interactions among 295
the components of the sensitivity calculation (data functionals L and Q; 296
solver S
1
m
; P), using the structure of the data vector to ensure eciency. 297
As we discuss next, a coarse parallelization (over transmitters, or unique for- 298
ward problems, similar to the approach used in Siripunvaraporn and Egbert 299
(2009)) is also implemented at the level of the sensitivity module, and is 300
thus completely isolated from details of the specic EM inverse problem that 301
ModEM is applied to. 302
303
4. Parallelization 304
ModEM incorporates a exible coarse-grained parallelization over forward 305
problems following the scheme of Meqbel (2009), a variant of the master- 306
16
worker parallelization method. The scheme minimizes communication be- 307
tween workers and master, and reduces memory requirements, by having 308
workers store results of forward computation for reuse in the solution of the 309
adjoint problem. Given the data vector transmitter/receiver structure, this 310
can be implemented quite easily, through parallelized versions of the up- 311
per level routines which organize Jacobian calculations (Jmult, JmultT and 312
calcJ in Table 2), also modied to allow for the bookkeeping necessary to 313
maintain storage eciencies in the worker nodes. Parallel versions of these 314
routines are contained in a separate module. 315
316
The scheme is implemented using calls to the standard Massage Pass- 317
ing Interface (MPI) communication library, with only complications arising 318
from eorts to minimize interaction of the parallelization with the rest of 319
the ModEM system. For example, messages passed between processors us- 320
ing MPI are restricted to standard data types in the programming language 321
used (i.e., in Fortran 95: Integer, Real, etc.), while ModEM makes extensive 322
use of derived data types, e.g., to store m, e and d as abstract encapsulated 323
objects. Thus, to allow a generic implementation of the parallelization which 324
can pass such objects around without reference to internal details of the ac- 325
tual data structure, it is necessary to implement specialized utility routines 326
for each specic instance of the Model Space, Solution Space and Data Space 327
modules. These are routines with generic names and interfaces that create 328
an MPI structured data type from the Fortran derived data type, thus allow- 329
ing communication routines in the MPI module to also treat these objects 330
abstractly while sending and receiving. The application specic MPI source 331
17
code for these utility routines is kept in separate les, which are included in 332
the appropriate serial modules when compiling for parallel execution. Finally 333
note that relatively trivial modications to each specic inversion routine is 334
required, essentially to route calls to forward modeling and Jacobian calcula- 335
tions through the parallel or serial versions of these routines, as appropriate. 336
To avoid having two copies of these routines this is accomplished through 337
compiler directives. 338
339
The general relationship of the MPI Main module to the rest of ModEM 340
is illustrated in Figure 1, and an overview of our MPI implementation is 341
given in Figure 2. Briey, after processor initialization, one processor is 342
assigned as the master, and the rest as workers. The workers enter a queue 343
(in routine Worker Job, say) and await messages from the master, while the 344
master steps through the actual inversion algorithm, until a step requiring 345
solution of the forward or adjoint problem is reached. At this point the 346
master distributes messages to all workers, indicating which task to perform, 347
and providing any necessary input data. The worker executes the requested 348
job, typically for a single transmitter, by calling the appropriate sensitivity 349
or interface layer subroutinesthe same code that would be executed in a 350
serial implementation. Thus, except for data type conversion, essentially all 351
MPI specic routines are hosted inside a single MPI module. 352
Our parallel scheme is designed to minimize communication between pro- 353
cessors. Consider for example the matrix-vector multiplication implemented 354
in JmultT. As noted in Table 2, this routine takes the background EM so- 355
lution (which typically will already have been computed to evaluate model 356
18

MPI_Main

Master:
:
:
Subroutine Master_Job_fwdPred (m0,d,e0)
1- Task = Distribute Model Parameters
-Send a copy of model parameters to all workers.
2- Task = Solve Forward Problem
- Send the index of one transmitter to each worker
- Receive the predicted data
End subroutine
:
Worker:
Do
Receive task from Master

if task = Distribute Model Parameters then
Receive a copy of model parameters from Master

if task = Solve Forward Problem then
Receive the index of one transmitter from Master
Call fwdPred_TX (m0,d,e0)
Send the predicted data for one transmitter to Master

if task = Stop then
EXIT loop

Loop


b)
a)
c)
Serial version: i.e. inside NLCG
:
:
:
:
:
:
Call fwdPred (m0,d,eAll)
:
:
:
Serial/Parallel version: i.e. inside NLCG
:
:
:
#if def=MPI then
Call Master_Job_fwdPred (m0,d,eAll)
#else
Call fwdPred (m0,d,eAll)
#end if
:
:
:

Figure 2: Pseudo-code for the Message Passing Interface (MPI) modular implementation.
19
mist) as an optional input. We keep the solution for a particular transmitter 357
on the processor used to compute it, and assign the same transmitter to this 358
processor for the task of computing the product J
T
d (for a single transmit- 359
ter). Thus it is not necessary to gather EM solutions for all transmitters on 360
the master processor and then re-distribute these to the workers. The master 361
sends only a transmitter index and the data vector (d) to each worker to per- 362
form the multiplication in parallel. Of course the resulting model parameters 363
must still be returned to the master to be summed to complete computation 364
of J
T
d for the full model vector. 365
5. Interface Layer 366
The central grouping in Figure 1 provides an interface between the generic 367
sensitivity and inversion modules discussed in the previous section, and the 368
implementation specic details of the numerical discretization modules. In 369
particular, the EM solution and source terms e and b are dened at this 370
level in the solution space module, as is the abstracted forward solver (which 371
maps b to e), the problem specic data functionals (which map e to d), 372
and the operator P, which denes the sensitivity of the forward solver to the 373
model parameter. All of these objects are used extensively by the generic 374
sensitivity and inversion routines, and thus must have standardized names 375
and interfaces, as in Table 2. 376
From the perspective of object oriented programming, the interface layer 377
denes specic instances of abstract classes, which implement the methods 378
needed by the generic top-level sensitivity routines. In addition to hiding 379
details of the actual numerical implementation from generic inversion rou- 380
20
R
o
u
t
i
n
e
N
a
m
e
I
n
p
u
t
s
O
u
t
p
u
t
s
U
s
e
d
F
o
r
M
o
d
u
l
e
i
n
i
t
S
o
l
v
e
r
i
T
x
,
m
e
0
(
e
,
b
)
f
w
d
P
r
e
d
,
J
m
u
l
t
,
J
m
u
l
t
T
,
c
a
l
c
J
F
o
r
w
a
r
d
S
o
l
v
e
r
(
D
r
i
v
e
r
)
e
x
i
t
S
o
l
v
e
r
e
0
(
e
,
b
)
f
w
d
P
r
e
d
,
J
m
u
l
t
,
J
m
u
l
t
T
,
c
a
l
c
J
F
o
r
w
a
r
d
S
o
l
v
e
r
(
D
r
i
v
e
r
)
f
w
d
S
o
l
v
e
r
i
T
x
(
F
W
D
o
r
A
D
J
,
b
)
e
f
w
d
P
r
e
d
,
J
m
u
l
t
,
J
m
u
l
t
T
,
c
a
l
c
J
F
o
r
w
a
r
d
S
o
l
v
e
r
(
D
r
i
v
e
r
)
P
m
u
l
t
e
0
,
m
0
,
m
b
J
m
u
l
t
S
o
l
v
e
r
S
e
n
s
i
t
i
v
i
t
i
e
s
P
m
u
l
t
T
e
0
,
m
0
,
e
m
(
m

)
J
m
u
l
t
T
,
c
a
l
c
J
S
o
l
v
e
r
S
e
n
s
i
t
i
v
i
t
i
e
s
Q
m
u
l
t
e
0
,
m
0
,
d
0
,
m
d
J
m
u
l
t
D
a
t
a
S
e
n
s
i
t
i
v
i
t
i
e
s
Q
m
u
l
t
T
e
0
,
m
0
,
d
m
(
m

)
J
m
u
l
t
T
D
a
t
a
S
e
n
s
i
t
i
v
i
t
i
e
s
L
m
u
l
t
e
0
,
m
0
,
d
0
,
e
d
J
m
u
l
t
D
a
t
a
S
e
n
s
i
t
i
v
i
t
i
e
s
L
m
u
l
t
T
e
0
,
m
0
,
d
b
J
m
u
l
t
T
D
a
t
a
S
e
n
s
i
t
i
v
i
t
i
e
s
Q
r
o
w
s
e
0
,
m
0
,
i
D
t
,
i
R
x
m
Q
m
u
l
t
,
Q
m
u
l
t
T
,
c
a
l
c
J
D
a
t
a
F
u
n
c
t
i
o
n
a
l
s
L
r
o
w
s
e
0
,
m
0
,
i
D
t
,
i
R
x
l
L
m
u
l
t
,
L
m
u
l
t
T
,
c
a
l
c
J
D
a
t
a
F
u
n
c
t
i
o
n
a
l
s
d
a
t
a
R
e
s
p
e
,
m
,
i
D
t
,
i
R
x
Z
f
w
d
P
r
e
d
D
a
t
a
F
u
n
c
t
i
o
n
a
l
s
f
w
d
P
r
e
d
m
,
d
0
d
(
e
)
P
a
r
a
l
l
e
l
I
n
v
e
r
s
i
o
n
A
l
g
o
r
i
t
h
m
s
J
a
c
o
b
i
a
n
C
o
m
p
u
t
a
t
i
o
n
s
c
a
l
c
J
m
0
,
d
0
J
P
a
r
a
l
l
e
l
I
n
v
e
r
s
i
o
n
A
l
g
o
r
i
t
h
m
s
J
a
c
o
b
i
a
n
C
o
m
p
u
t
a
t
i
o
n
s
J
m
u
l
t

m
,
m
0
,
d
0
(
e
)

d
P
a
r
a
l
l
e
l
I
n
v
e
r
s
i
o
n
A
l
g
o
r
i
t
h
m
s
J
a
c
o
b
i
a
n
C
o
m
p
u
t
a
t
i
o
n
s
J
m
u
l
t
T
m
0
,

d
(
e
)

m
P
a
r
a
l
l
e
l
I
n
v
e
r
s
i
o
n
A
l
g
o
r
i
t
h
m
s
J
a
c
o
b
i
a
n
C
o
m
p
u
t
a
t
i
o
n
s
T
a
b
l
e
2
:
P
u
b
l
i
c
R
o
u
t
i
n
e
s
U
s
e
d
f
o
r
S
e
n
s
i
t
i
v
i
t
y
C
o
m
p
u
t
a
t
i
o
n
s
.
C
o
n
v
e
n
t
i
o
n
s
u
s
e
d
i
n
c
a
l
l
-
i
n
g
a
r
g
u
m
e
n
t
s
:
i
D
t
=
d
a
t
a
t
y
p
e
;
i
R
x
=
r
e
c
e
i
v
e
r
;
i
T
x
=
t
r
a
n
s
m
i
t
t
e
r
.
O
t
h
e
r
s
y
m
b
o
l
s
a
r
e
a
s
i
n
T
a
b
l
e
1
,
w
i
t
h
s
u
b
s
c
r
i
p
t
z
e
r
o
s
d
e
n
o
t
i
n
g
b
a
c
k
g
r
o
u
n
d
m
o
d
e
l
p
a
r
a
m
e
t
e
r
s
a
n
d
s
o
l
u
t
i
o
n
v
e
c
t
o
r
s
u
s
e
d
f
o
r
l
i
n
e
a
r
i
z
a
t
i
o
n
,
a
n
d
f
o
r
t
h
e
t
e
m
p
l
a
t
e
d
a
t
a
v
e
c
t
o
r
.
T
h
e
t
e
m
p
l
a
t
e
d
a
t
a
v
e
c
t
o
r
d
0
g
e
t
s
o
v
e
r
w
r
i
t
t
e
n
w
i
t
h
t
h
e
c
o
m
p
u
t
e
d
v
e
c
t
o
r
d
o
n
o
u
t
p
u
t
,
w
h
e
r
e
a
p
p
r
o
p
r
i
a
t
e
.
Z
d
e
n
o
t
e
s
a
n
a
r
r
a
y
o
f
r
e
a
l
r
e
s
p
o
n
s
e
s
.
O
p
t
i
o
n
a
l
i
n
p
u
t
s
o
r
o
u
t
p
u
t
s
a
r
e
i
n
p
a
r
e
n
t
h
e
s
e
s
;
m

i
s
t
h
e
o
p
t
i
o
n
a
l

i
m
a
g
i
n
a
r
y

c
o
m
p
o
n
e
n
t
o
f
t
h
e
m
o
d
e
l
p
a
r
a
m
e
t
e
r
o
u
t
p
u
t
b
y
P
m
u
l
t
T
.
21
tines, the interface layer is where source and receiver details for specic EM 381
applications are dened. As we shall see in Section 7, inversions for dierent 382
EM methods (e.g., MT and CSEM) may be developed using the same base 383
of numerical discretization modules and the same generic inversion modules 384
through modications to the interface layer. 385
386
Central to the interface layer is the solution space module, which denes 387
derived data types used to represent the solution vector e and source vector 388
b of the discrete forward problem (2) for a single transmitter. These two 389
distinct data types represent objects which are eectively in the same space, 390
but we have found it useful to distinguish between vectors used for sources 391
and solutions. The sparse vector data type is also used to represent elements 392
of this same space, in this case to provide a storage ecient representation 393
of the sparse data functionals which constitute rows of the operator L. Basic 394
methods for manipulating all of these objects, including creation, destruction, 395
I/O, copying, linear algebra and dot-product operations, are used extensively 396
throughout the Jacobian computations and thus must be implemented with 397
standardized names and interfaces. 398
Note that the full solution vector employed in an inverse problem will 399
in general be an array of solution vector objects, one for each transmitter. 400
Elements of this array could in principal be quite dierent, for example in 401
the case of joint inversion. For instance, joint inversion of 3D MT and CSEM 402
could be easily accommodated by storing a solution for the MT problem in 403
e(1) (encapsulating solutions for two source polarizations for one period), 404
while e(2) could represent a solution for a single controlled source transmit- 405
22
ter. And, for the 2D MT problem, the array of solution vector objects would 406
generally contain both TE mode (electric eld) and TM mode (magnetic 407
eld) forward solutions. The interface layer hides such complications from 408
the generic sensitivity and inversion routines, though obviously these details 409
are critical to implementation of actual computations. 410
411
5.1. Forward Solver 412
The purpose of this module is to provide a uniform interface between for- 413
ward modeling routines implemented in the numerical discretization layer, 414
and the generic inversion and sensitivity routines of Section 3. The key public 415
routines in this module include initialization (initSolver), deallocation and 416
clean up (exitSolver), and the actual solver (fwdSolver), with interfaces 417
as dened in Table 2. Internal functioning of these routines is application 418
dependent, and controlled by the index into the transmitter dictionary, iTx. 419
For example, in the 3D MT case (see Section 7) the output of fwdSolver e 420
consists of solutions for two source polarizations, requiring separate calls to 421
the actual numerical solver with dierent boundary conditions, for the fre- 422
quency dened by iTx. In the 2D MT case only a single solution is required, 423
but now there are two possibilities: either a TE or a TM solver might be 424
required, depending on the mode, which again would be dened (along with 425
the frequency) with reference to the transmitter dictionary. While applica- 426
tions may dier in internal function (a loop over polarizations for the 3D 427
MT implementation vs. a case statement for the 2D case), routine names, 428
interfaces, and abstract functionality (i.e., compute e for transmitter iTx) 429
are standardized. 430
23
Initialization (initSolver) is explicitly separated from actual solution 431
to enable more ecient computational strategies. For example, if a direct 432
matrix LU factorization is practical, this can be implemented in the initial- 433
ization routine, and executed only if the coecient matrix has changed from 434
the previous solver call. More generally, repetition of operator setup steps 435
can also be minimized by keeping track of attributes (frequency, conductivity 436
parameter) used for the previous solution. The responsibility for these e- 437
ciencies lies with the initialization routinehigher level routines that require 438
solutions of the governing PDE always call the initialization routine rst. 439
The cleanup routine exitSolver is only called when it is desired to deallo- 440
cate all operator arrays and return solver module data to the state it had 441
before the rst call to initSolver. Finally, fwdSolver implements the gen- 442
eral solver, allowing arbitrary forcing and boundary conditions, and solutions 443
for both the usual forward problem and its transpose, or adjoint. By default, 444
forcing for the forward problem is computed internally, with reference to the 445
transmitter dictionary. For sensitivity calculations the forcing is provided as 446
an optional argument, along with an optional switch that species execution 447
of the adjoint or forward solver. 448
5.2. Data Functionals 449
This module implements the data functionals (
j
(e, m) in Table 1), as 450
well as the linearizations with respect to e (l
j
; rows of L) and, if necessary, 451
m (q
j
; rows of Q). As shown explicitly in EK12, these functionals can be 452
expressed in terms of (1) a mapping which converts the primary eld (e.g., 453
electric) to the dual eld (e.g., magnetic); (2) interpolation operators for the 454
primary and dual elds, which simulate the process of measuring EM compo- 455
24
nents at specic locations; (3) functionals of the measured eld components, 456
e.g., an impedance or apparent resistivity (see Eqs. 10 and 11). The rst two 457
components are tied closely to the numerical grid, and are implemented at 458
the level of the numerical discretization, to be discussed in Section 6. These 459
decompositions of the linearized data functionals into a set of elementary 460
mappings allows for a completely streamlined modular implementation of the 461
data functionals, making it especially easy to write the sensitivity derivations 462
for any general data functional in terms of pre-existing elementary operators. 463
464
Three public routines with standardized names and interfaces (see Ta- 465
ble 2) implement data functionals for use in sensitivity and inversion com- 466
putations. dataResp evaluates the (possibly non-linear) data functionals 467

j
(e, m) for a single arbitrary receiver/data type. Information about the 468
data type (e.g., an impedance, apparent resistivity or phase) is obtained 469
by reference to the data type dictionary (indexed by iDt) while meta-data 470
such as the site location is obtained from the receiver dictionary (indexed 471
by iRt). Through calls to lower level (numerical implementation specic) 472
routines all necessary interpolation operators are created, and magnetic and 473
electric components at the observation location are evaluated. Any addi- 474
tional computations (e.g., impedance, amplitude, phase) are completed at 475
the interface layer. Note that in general multiple components (e.g., all ele- 476
ments of an impedance tensor) may be returned for some data types. Note 477
also that all data functional routines implement computations (selected by a 478
case statement) for the full range of data types and transmitters. 479
Lrows implements the linearization of the data functional with respect to 480
25
variations in the EM solution e. In the simplest case, where the data is just a 481
measured eld component (as for example in CSEM methods) and
j
(e, m) 482
is already linear in e, this routine simply computes the appropriate evaluation 483
functional (exactly as in dataResp), and returns this, packaged as a sparse 484
vector in the EM solution space. For the more general case the linearized 485
functionals l
j
are linear combinations of those used to represent basic eld 486
component evaluation, with coecients that depend on the background EM 487
solution e
0
(see Eq. (10). These sparse vectors are used in Lmult (Table 2) to 488
form the dot products with e required to assemble the data vector object d = 489
Le. LmultT multiplies these vectors by the appropriate data components, 490
and sums to form the forcing for the adjoint system L
T
d. See Section 7 for 491
a specic example (3D MT). 492
The function of Qrows is similar, but instead of data functionals this rou- 493
tine returns one or more model parameters, q
j
. A general expression for these 494
rows of Q is given in (11), with further details provided in EK12, Sec. 5.2.3. 495
Here we note only that each of these model parameters can be expressed as 496
a product of a sparse vector in the EM solution space (formed as for l
j
with 497
reference to the data type and receiver dictionaries), and the (transposed) 498
linearized model parameter mapping
T
m
0
(see Section 6). Although in many 499
cases the result will be sparse in the model parameter space, there are ex- 500
ceptions, so ModEM represents the q
j
as full model parameters. Also, note 501
that while model parameters are assumed to be real, the sparse vectors in the 502
EM solution space that they multiply are complex. As discussed in EK12, in 503
some situations (e.g., computation of the full Jacobian) both real and imag- 504
inary parts are required, while for other situations (e.g., computation of the 505
26
data mist derivative) only the real part is required. In Qmult (Table 2), dot 506
products are formed with the input model parameter and assembled into the 507
data vector object d = Qm. For the transpose QmultT, the model param- 508
eters q
j
are multiplied by the respective data components and summed to 509
obtain m = Q
T
d. 510
511
5.3. Solver Sensitivity 512
The nal component of the interface layer implements P, which eec- 513
tively denes the sensitivity of the forward problem coecient matrix S
m
to 514
the model parametrization (see eq.(5)). As seen from (8) this operator can 515
be divided into two components: Udiag(Ve
0
), which depends explicitly on 516
the problem formulation (i.e., the form of the PDE used) and the numerical 517
implementation, but not the specic model parametrization, and
m
0
, the 518
linearized mapping of the model parameter to the grid. Pmult computes the 519
product Pm by rst applying
m
0
to model parameter object m, and then 520
applying (Udiag(Ve
0
)) to the result. This routine returns a source vector 521
object, of the proper type to provide forcing for the solver in a computation 522
such as Jm = LS
1
Pm. PmultT multiplies a solution vector object by P
T
, 523
reversing these steps, and resulting in a model parameter vector. As with 524
QmultT, the output of PmultT is intrinsically complex, and in some cases 525
(e.g., computing the full sensitivity matrix) both real and imaginary parts 526
are saved. 527
528
Therefore, the decomposition of the solver sensitivity operator P in Eq. 8 529
into the elementary components of the forward problem, identied in Eq. 7, 530
27
simplies implementation of sensitivity computations for a wide range of 531
EM problems. Once the forward solver is theoretically decomposed into a 532
combination of elementary discrete operators, the solver sensitivities may be 533
immediately expressed in terms of these elementary components and plugged 534
into the inversion scheme to provide the correct discrete numerical derivative. 535
536
6. Numerical Discretization 537
The nal group of modules (Figure 1) provide the basic numerical grid, 538
structures which represent primary and secondary elds, the actual numerical 539
forward solver, and a basic set of interpolation functionals for point evalua- 540
tion of electric and magnetic elds. For purposes of our discussion here we 541
also include the model parameter module in this group, although, as dis- 542
cussed in Section 3, these can also be considered higher level objects, which 543
are manipulated directly by inversion and sensitivity routines, and thus must 544
conform to generic ModEM interface standards. However, even though the 545
internal structure of m (and the model covariance) can be quite independent 546
of the grid and other numerical implementation details, the model parameter 547
does play a critical role in dening the discrete numerical operator and thus 548
must interact quite strongly with other numerical discretization components. 549
The remaining numerical discretization modules have no direct interac- 550
tion with the generic inversion and sensitivity routines, so a wide range of 551
specic implementations can be accommodated through modications at the 552
interface layer. It is thus most useful to discuss these modules in the context 553
of a specic example. We do this in Section 7, after oering some general 554
28
guidelines on those features, conformance to which ensures the functionality 555
required by more generic components, and an overview of how modules can 556
be organized (e.g., as outlined in Figure 1) to simplify interfacing with the 557
rest of ModEM. 558
The basic numerical discretization data objects are the grid, and struc- 559
tures which represent the discretized electric and/or magnetic elds. In the 560
terminology of EK12, the latter are elements of the primary and dual spaces, 561
S
P
and S
D
. Although not strictly required, explicitly dening an EM eld 562
data type (here the type name is unimportant), along with the related linear 563
algebra and other basic data object operations, greatly simplies construc- 564
tion of the required encapsulated solution vector and source vector objects 565
(e and b) discussed in Section 5. Similarly, dening sparse vector represen- 566
tations for elements of S
P
simplies construction of data functionals. An 567
example of an EM eld module, including relevant interactions with higher 568
level modules, is given in Section 7. An encapsulated data structure which 569
denes the numerical grid does have to be dened, although there are no 570
specic requirements on the form that this takes. Some routines in the sen- 571
sitivity module pass pointers to the grid, e.g., to allow data site locations to 572
be referenced to the numerical solution grid. 573
Forward modeling, is of course the core of the inversion. In principal an 574
existing forward code can be used, but to be useful for ModEM several con- 575
ditions must be met. First, the solver should be general, in the sense that the 576
solution can be computed correctly for arbitrary sources and boundary data, 577
even if the forcings encountered for the forward problem are more restricted. 578
For example, for the usual MT forward problem forcing is restricted to the 579
29
boundaries, but more general source terms must be allowed for to compute 580
sensitivities. Second, capability to solve the transposed system is required. 581
Because the EM equations are essentially self-adjoint, forward codes can gen- 582
erally be easily amended to implement adjoint calculations (e.g., see EK12 583
(Sec. 3), and Section 7), although there can be some subtle issues (e.g., Kel- 584
bert et al. (2008)). Finally, the solver should have a clean interface that 585
allows interaction with higher level routines. At a minimum these must in- 586
clude initialization of the solver, and updating the PDE coecients (which 587
depend on the model parameter, and the frequency), as well as actually solv- 588
ing the forward or transposed equations for a particular set of sources and 589
boundary conditions. 590
591
Developing a separate module with standardized functionality for the in- 592
terpolation functionals used to evaluate electric and magnetic elds at an 593
arbitrary point within the model domain (e.g., EM eld interpolation in Fig- 594
ure 1) greatly simplies implementation of data functionals at the interface 595
level (see Sec 5). The essential requirement is for routines which compute 596
interpolation functionals for both primary and secondary eld components, 597
and return these as sparse vectors in the primary eld space S
P
. The case of 598
the primary (e.g., electric) eld, is clear; the non-zero components of are 599
the local interpolation weights. For the dual eld (e.g., the magnetic eld 600
h = Te), must incorporate the transformation operator (T) used to com- 601
pute the dual eld, so that the functional can still be applied directly to the 602
primary eld vector. Note that it is not necessary to form the full transfor- 603
mation operator, as only a few components are needed to form the dual eld 604
30
evaluation functional for any location. If either interpolation functionals, or 605
the transformation operator, depend explicitly on the model parameter some 606
additional sparse vectors are required for assembly of rows of Q. Further 607
details are provided in EK12 (Sec. 5.2.3). 608
The sparse vectors returned by EM eld interpolation routines are the ba- 609
sic building blocks for the non-linear data functionals
j
(e.g., impedances), 610
as well as the corresponding linearizations the rows of L and Q discussed in 611
Section 5. The evaluation functionals of course depend critically on details 612
of the numerical grid, while computation of something like an impedance or 613
apparent resistivity from the interpolated electric and magnetic eld com- 614
ponents does not. Thus, a dierent numerical discretization of the problem 615
(e.g., nite elements with a non-structured grid) would require dierent in- 616
terpolation routines. However, if these were again represented as sparse 617
vectors in the appropriate discrete EM eld spaces, higher level data func- 618
tional routines (e.g., for calculation of impedance elements), or construction 619
of the corresponding linearized data functionals, would remain unchanged. 620
Conversely, to add new data types (e.g., inter-station magnetic transfer func- 621
tions) there is no need to revisit the interpolation aspects of the problem. 622
623
The model parameter module consists of three clearly distinguishable 624
components: (1) an encapsulated data structure which represents m, to- 625
gether with methods to implement vector space operators, including dot 626
products; (2) the model parameter covariance or regularization operator; (3) 627
mappings between the model parameter and the numerical grid. Note that 628
to some extent these components can be modied independently e.g., a dif- 629
31
ferent covariance implementation can be used without changes to the other 630
two components, and if the numerical discretization were changed, only the 631
mapping component would have to be modied. Key aspects of the rst two 632
components have been sketched in Section 3. The third component denes 633
three routines that interact with the interface layer. The rst denes (m), 634
the model parameter mapping which species resistivity or conductivity on 635
the numerical grid, as required to dene the forward operator. This will 636
generally be called by the forward solver initialization routine (initSolver). 637
The second and third implement multiplication by the Jacobian of the model 638
parameter mapping = /m, and its adjoint
T
. These are required to 639
implement multiplication by P and Q, and their adjoints. To the extent that 640
the basic interpolation functionals depend on the model parameter, it is also 641
useful to dene the restriction of (m) to specic grid elements. 642
643
7. Example: 3D EM 644
7.1. Numerical Discretization 645
As specic examples, we consider application of ModEM to 3D MT and 646
CSEM inverse problems. Both are implemented using the same numerical 647
discretization modules, based on nite dierence (FD) solution of the quasi- 648
static Maxwells equations in the frequency domain, in the form given by (9) 649
on a staggered-grid (e.g., Yee (1966); Siripunvaraporn et al. (2002)). 650
We begin with a summary of the discretization of the equations, following 651
the notation of EK12 where further details are provided. Electric elds are 652
dened on cell edges (see Figure 3); the (primary) space of such discrete 653
32
Figure 3: Staggered nite dierence grid for the 3D MT forward problem. Electric eld
components dened on cell edges are the primary EM eld component, which the PDE is
formulated in terms of. The magnetic eld components can be dened naturally on the
cell faces; these are the secondary EM eld in this numerical formulation.
33
vector elds is denoted S
P
. The dual space, denoted by S
D
consists of vector 654
elds with components dened on cell faces, representing magnetic elds. 655
Equation (9) may then be written in discrete form as 656
_
C

C+ diag(i(m))

e = j
s
(12)
Be = e
b
, (13)
where e S
P
, j
s
S
P
is the optional interior forcing, C : S
P
S
D
is 657
the discrete approximation of the curl operator mapping cell edge vectors 658
(including boundaries) to interior cell face vectors, C

: S
D
S
P
is the dis- 659
crete curl mapping interior cell face vectors to interior cell edges, B extracts 660
boundary edges, and e
b
gives the specied boundary data. The interior forc- 661
ing on the right hand side of (12) is applicable to CSEM but not MT, and is 662
dened on the interior primary cell edges only. The dependence of the oper- 663
ator coecients on the model parameter is made explicit in (12) through the 664
mapping : M S
P
, where M is the model parameter space. Magnetic 665
elds corresponding to an electric eld solution e can be expressed as 666
h = (i)
1
Ce = Te. (14)
To implement numerical discretization modules based on this formula- 667
tion, we dene the basic EM eld object as a derived data type containing 668
three separate 3D arrays (x, y, and z components), together with a pointer 669
to the underlying basic grid (which denes the geometry and georeference 670
of the staggered grid), and a tag to indicate if the vector eld is dened 671
on the primary (edges) or dual (faces) grid. Several variants on these basic 672
34
complex vector objects are also dened, including scalar elds (dened either 673
on cell centers or nodes), and allowing for real as well as complex versions 674
of both scalars and vectors. For example, complex scalars are used to rep- 675
resent electric potentials (used in the solver for the divergence correction; 676
Smith (1996)); a real vector can be used to represent electrical conductiv- 677
ity dened on cell edges, i.e., (m) in (9). We also dene a sparse vector 678
representation for elements of S
P
and S
D
, to allow ecient representation 679
and storage of measurement functionals. For all of the variants of the basic 680
vector and scalar elds there are routines for creation, deallocation, linear 681
algebra, dot products, and point-wise multiplication. Algebraic routines for 682
relevant combinations of types (e.g., dot products of sparse and full vectors, 683
used for evaluation of eld component functionals; point-wise multiplication 684
of complex and real vector elds, required to compute iE) are also in- 685
cluded. Finally, we also dene a specialized data structure to store data for 686
boundary conditions. 687
688
The forward operator of (9) is implemented using a matrix-free approach, 689
and the system of linear equations is solved iteratively with a quasi-minimum 690
residual (QMR) scheme, with a level-1 incomplete LU decomposition for 691
pre-conditioning. As in Smith (1996) and Siripunvaraporn et al. (2002), a 692
divergence correction is applied periodically, with the Poisson-like equation 693
solved with pre-conditioned conjugate gradients. Everything is coded in an 694
object-oriented manner, so that, for example, solvers and pre-conditioners 695
are stand alone modules which can be easily replaced with alternative algo- 696
rithms. Implementation of the transposed solver is straightforward. Follow- 697
35
ing EK12 (Sec. 3 and Appendix D) the dierential operator S
m
= C

C + 698
diag(i(m)) satises S
m
T
= VS
m
V
1
where V is the diagonal matrix 699
of integration volume elements for S
P
. This means that VS
m
= S
m
T
V is 700
symmetric, and to improve performance of the QMR solver we transform to 701
this modied system. Thus, to solve S
m
e = b we multiply b by V before 702
calling the solver; for solutions to S
m
T
e = b this preliminary step is replaced 703
by multiplying the output of the solver by V
1
. 704
705
Interpolation functionals are based on tri-linear splines. Separate routines 706
compute interpolation coecients for vectors dened on edges and faces, re- 707
turning sparse vectors
P
S
P
and
D
S
D
respectively. A third routine 708
combines the coecients of
D
with appropriate rows of the transformation 709
operator of (14) to construct the sparse magnetic eld interpolation func- 710
tional, represented as a sparse vector in S
P
. 711
712
For an initial implementation of the model parameter module, we have 713
used a very simple, and classical, scheme: Mconsists of the space of log con- 714
ductivities dened independently on each of the cells in the underlying numer- 715
ical grid. Thus, as discussed in EK12, a physically consistent (current con- 716
serving) form for the mapping : M S
P
is given by (m) = Wexp(m), 717
where W is the matrix representing the operator that assigns the weighted 718
average of the four surrounding cells to each edge. The Jacobian of this 719
operator evaluated at the background conductivity m
0
is thus 720

m
0
= W[ diag(exp(m
0
))]. (15)
36
We also require the transpose
T
m
0
, a mapping from cell edges to cells, com- 721
puted as a weighted sum of contributions from all edges that bound a cell. 722
Our basic model covariance is somewhat more novel: we have imple- 723
mented a recursive autoregressive covariance smoother, loosely based on 724
Purser et al. (2003a,b), which penalizes deviations from a specied prior 725
model conductivity. Horizontal and vertical smoothings may be dierent; 726
moreover, the extent of horizontal smoothing may vary with depth. The 727
conguration le also allows the user to divide the model domain into dis- 728
joint pieces, allowing smoothing to be turned o across domain boundaries, 729
and also allowing model parameters within a domain to be frozen. It is 730
worth stressing again that the model parametrization and covariance are 731
completely independent of the rest of ModEM. In practice, we envision sup- 732
porting a menu of model parametrization and covariance choices for 3D EM 733
problems. 734
735
7.2. Interface Layer 736
The interface layer is slightly dierent for MT and CSEM implemen- 737
tations, starting with diering requirements for the solution vector e. As 738
discussed in detail in EK12, to allow evaluation of the basic 3D MT data 739
type, the impedance tensor, e must contain a pair of complex vector EM 740
eld structures, corresponding to solutions for two independent source po- 741
larizations. Similarly, the source term b in (2) must dene boundary condi- 742
tions for two forward problems, and the linearized data functionals (e.g., l
j
) 743
must represent a pair of sparse elements of S
P
, since calculating a predicted 744
impedance requires sampling solutions for both polarizations. On the other 745
37
hand, for most CSEM problems each observation will correspond to a single 746
known source, so all interface layer objects (i.e., e, b, l
j
) will contain only 747
a single EM eld structure. To accommodate both of these cases we have 748
dened solution vector and related data types for 3D EM problems to allow 749
for any number of coupled source polarizations, dened by a variable (e.g., 750
nPol ) in the data structure. This makes treatment of a joint MT/CSEM 751
problem straightforward e.g., the array of solution vectors e can contain 752
solutions for both sorts of EM data, with some having nPol =1, and some 753
nPol =2, with the appropriate case selected by reference to the transmitter 754
dictionary. 755
756
The interface layer Forward Solver module manages forward and adjoint 757
solutions, including setting up sources and/or boundary conditions for the 758
forward problem. This module thus also must provide slightly dierent func- 759
tionality for the MT and CSEM problems. For the MT forward problem, 760
each call to fwdSolver requires two calls to the FD solver with boundary 761
conditions for two source polarizations (e.g., computed by solving 2D forward 762
problems for the appropriate frequency, dened by iTx). Adjoint solutions 763
similarly require a pair of FD solver calls (now boundary conditions in both 764
cases are homogeneous, but sources will dier). Of course only a single FD 765
solver call will generally be required in the CSEM case, both for forward and 766
for adjoint problems. A more signicant dierence is that for many CSEM 767
problems (e.g., with point dipole sources) it is natural to use a secondary eld 768
formulation (e.g., Alumbaugh et al., 1996) for the forward problem. This can 769
be quite simply implemented with changes only to the interface level, using 770
38
a support module which solves the forward problem for a 1D background 771
conductivity semi-analytically. This is used in the CSEM version of the For- 772
ward Solver module, where routine initSolver sets up the background and 773
anomalous conductivity and does preliminary computations for the 1D so- 774
lution (using the 1D CSEM forward solver of Key (2009)), with all results 775
stored as private saved variables. Then, on each call to fwdSolver the appro- 776
priate location and frequency is extracted from the transmitter dictionary, 777
and the 1D background solution for this transmitter, and then the forcing 778
for the secondary eld solution, are computed. The FD solver is then called 779
to compute the 3D secondary eld, which is added to the 1D background 780
solution. The total eld solution is returned by fwdSolver. All of these dif- 781
ferences in implementation are hidden, both from higher level sensitivity and 782
inversion routines, and from the lower level FD codes. Again, it is straight- 783
forward to allow for joint inversion, combining the functionality required for 784
MT and CSEM into single module. 785
786
For a simple CSEM problem, where data are direct observations of eld 787
components, implementation of data functionals is trivial at the interface 788
level, as the essential functionality is already provided by the basic inter- 789
polation functionals. In this case Lrows is just a wrapper which calls the 790
appropriate interpolation functional (determined by the receiver dictionary), 791
and packages the output as a sparse vector. The forward routine dataResp 792
is also simple, but goes one step further to form the dot product between 793
the sparse vector and the solution vector, returning predicted data. For 3D 794
MT all data types are functions of impedance tensor elements, which are 795
39
simple non-linear functions of the raw eld components. Routine dataResp 796
thus rst computes the basic interpolation functionals required to evaluate 797
E
x
, E
y
, H
x
, and H
y
at the site, and applies these to solutions for both 798
polarizations. Then the predicted impedance tensor can be computed, fol- 799
lowed (if needed) by transformations required for a particular data type (e.g., 800
computation of an apparent resistivity or phase). Multiple data types are 801
supported, with appropriate code selected by a case statement. Lrows im- 802
plements linearizations of these calculations, forming linear combinations of 803
the basic interpolation functionals, following specic formulae given in EK12. 804
Neither the tri-linear spline interpolation coecients (
P
and
D
as in Ap- 805
pendix 3) nor the transformation operator T dened in (14) depend directly 806
on m, so Q 0. Qrows thus simply outputs q
j
= 0, for the MT and CSEM 807
implementations discussed here. 808
809
Finally, the form for P can be easily derived using results from EK12, 810
summarized in Section 2. Comparing (12) with (7), and using (8) and (15), 811
with the identications S
0
C

C, U iI, V I we nd the formal 812

expression 813
P = diag(ie
0
)
m
0
= diag(ie
0
) W[ diag(exp(m
0
))], (16)
where W maps from the model parameter to the primary grid. Note how- 814
ever, that for the MT case the background solution vector e
0
= (e
0,1
, e
0,2
) 815
contains FD solutions for two source polarizations, and the product Pm 816
computed by Pmult actually results in a source vector object which en- 817
capsulates the forcing for a pair of forward problems, computed by rst 818
40
evaluating W[ diag(exp(m
0
))]m followed by pointwise multiplication by 819
ie
0,k
, k = 1, 2. For PmultT, the input is a solution vector e = (e
1
, e
2
). 820
Now we rst compute i(diag[e
0,1
]e
1
+diag[e
0,2
]e
2
), and then apply the ad- 821
joint model parameter mapping
T
m
0
, resulting in a model parameter object. 822
Implementation of these Solver Sensitivity routines is basically the same for 823
the CSEM case, but with the simplication of a single polarization in source 824
and solution vectors. 825
826
7.3. Application to 3D MT and CSEM problems 827
We illustrate some basic capabilities of ModEM by using the same Nu- 828
merical Discretization modules to set up a single synthetic example for 3D 829
MT and 3D CSEM problems and illustrate the relative resolution of the two 830
methods. 831
832
Our synthetic model for land MT and CSEM is loosely inspired by the 833
Commer and Newman (2009) marine synthetic example in that it is a verti- 834
cal superposition of horizontally folded structures, designed to showcase the 835
relative strengths and weaknesses of the two methods. It also mimics the 836
kind of contrasts encountered in a real land electromagnetic survey setting. 837
In this 3D model, the synthetic upper and lower conductive folds extend 838
along the strike, while the thin resistive structure is fully three-dimensional, 839
hosted in the upper fold with a length of 3 km along the strike. 840
The model domain is discretized into 50 50 50 cells, with horizontal 841
cell dimensions of 500 m
2
, logarithmically increasing towards the edges. Cell 842
thicknesses are 50 m in the upper 1000 m of the model, then increasing log- 843
41
arithmically downwards by a factor of 1.2, to the depth of 160 km. 844
845
The full MT impedance tensor is sampled at 182 sites (Figure 4; black, 846
yellow and green circles) distributed on a regular 2D array with site spacing 847
of 1 km in both x and y directions. The data are computed at 17 periods, 848
distributed logarithmically between 0.1 s to 1000 s, with 4 periods per decade. 849
An error oor of 2% of |Z
xy
Z
yx
|
1/2
is assigned to the full impedance tensor. 850
A dense receiver spacing is assumed for controlled-source EM (Figure 4; 851
500 m spacing, denoted by red circles), including a subset of the MT sites 852
(yellow circles) located in the vicinity of the shallow resistive block which 853
also serve as receivers (RX) for CSEM. 854
The CSEM data are computed using 28 electrical point-dipole transmit- 855
ters (TX; denoted by green circles in Figure 4) operated at 5 periods (0.1, 856
0.5, 1, 5 and 10 secs). The in-line (E
y
) component of the electric eld is 857
computed at each of the receivers. An error oor of 2% of |E
y
| is assigned to 858
the noise free E
y
data. 859
860
For 3D inversion of the MT data, we used the ModEM version of NLCG 861
(Figure A.7). A homogenous model with 100 m resistivity was used as 862
prior and starting model. In total, 25 NLCG iterations were required to 863
decrease the nRMS value (normalized Root Mean Square) from 23 down 864
to 1.01. The computation was parallelized over 35 Intel processors, resulting 865
in the run time for each NLCG iteration (comprising 2 forward modeling and 866
1 gradient computation) of 2 minutes. 867
The 3D CSEM inversion also used a homogeneous 100 m halfspace for 868
42
prior and starting models. We found that 325 NLCG iterations were required 869
to decrease the nRMS from 101 to 2.5, using 141 processors in parallel. 870
871
Results for the 2 cases are presented in Figure 5, clearly illustrating the 872
relative inability of the MT method to resolve this resistive layer. The shal- 873
low resistor is much clearer in the CSEM inversion. Conversely, MT is better 874
at imaging the shape of the deeper conductor even in the middle of the array, 875
where the MT site locations are sparser compared to the CSEM receiver cov- 876
erage. We conclude that a joint MT-CSEM inversion would be advantageous 877
to harness the relative strengths of the two methods. 878
As can be seen from the convergence summary plots (Figure 6), even 879
when the same synthetic model is employed and the same NLCG inversion 880
procedure applied for MT and for CSEM congurations, convergence behav- 881
ior is very dierent for these two problems. This illustrative example shows 882
that in a practical joint MT and CSEM inversion, some care is require to 883
balance ts to the two data types. Although implementation of joint MT 884
and CSEM inversion is in principle straightforward in ModEM (Meqbel and 885
Ritter, 2013), it is outside the scope of this paper. 886
887
8. Discussion and Conclusions 888
ModEM has already proved to be a capable and ecient program for 889
practical 3D MT inversion (Kelbert et al., 2012; Tietze and Ritter, 2013; 890
Meqbel et al., 2013). However, the real power of ModEM lies in its potential 891
as a tool for research, and for rapid development of new applications. Trans- 892
43
-8
-4
0
4
8
X
-
D
i
s
t
a
n
c
e

(
k
m
)
-8 -4 0 4 8
Y-Distance (km)
Field Setup:
MT sites
CSEM sites
MT/CSEM Sites
CSEM Transmitters
Figure 4: The eld setups used to compute the synthetic data. Black dots correspond
to MT sites and red dots to CSEM receiver locations (RX), placed in the vicinity of the
shallow resistive block, yellow dots serve as both MT and CSEM receivers. Green dots
correspond to the 28 electrical point dipole transmitters (TX) used to compute the CSEM
model responses; they also double as MT sites and CSEM receivers. There are 182 MT
sites and 325 CSEM receivers, total.
44
1.0
2.0
3.0
4.0
5.0
6.0
7.0
D
e
p
t
h

(
k
m
)
-5 -2.5 0 2.5 5
Y-Distance (km)
1 10 100 1000
Resistivity (Ohm.m)
1.0
2.0
3.0
4.0
5.0
6.0
7.0
D
e
p
t
h

(
k
m
)
-5 -2.5 0 2.5 5
Y-Distance (km)
1.0
2.0
3.0
4.0
5.0
6.0
7.0
D
e
p
t
h

(
k
m
)
-5 -2.5 0 2.5 5
Y-Distance (km)
True Model MT CSEM
Model slices at X=0 km
Figure 5: Center cross-sections of the inversion results of tting the MT (middle column)
and CSEM (right column) synthetic data. For comparison purposes the true model is
plotted on the left.
Figure 6: NLCG algorithm convergence for MT and CSEM congurations. Both the total
penalty functional (logarithmic scale in black) and the RMS value (linear scale in red) are
plotted. Note the dierences in both the run times and magnitudes, and overall behavior
(steeper and smoother for the MT problem, likely because the same regularization has a
stronger eect on MT than on CSEM).
45
lating pseudo-code of the sort shown in Figures A.7 and A.8 into a ModEM 893
inversion module is simple, since all required operators and data objects are 894
readily available. Thus, it would be easy to replace the NLCG scheme we 895
have implemented with quasi-Newton (e.g., Newman and Boggs, 2004), or 896
standard Gauss-Newton variants such as OCCAM (Constable et al., 1987), 897
Siripunvaraporn et al. (2005), as well as more novel schemes such as the 898
multi-frequency hybrid inversion of Egbert (2012). ModEM thus provides a 899
natural test-bed for development, renement, and comparison of inversion 900
search algorithms. Similarly, ModEM oers potentially great exibility in 901
dening the model parametrization and regularization, simplifying new de- 902
velopments and extensions in this critical direction, e.g., to allow bounds on 903
conductivities (Avdeev and Avdeeva, 2009), to allow simultaneous inversion 904
for near-surface distortion parameters (e.g., DeGroot-Hedlin and Constable, 905
2004), or to decouple the model parameter and numerical solver grids (e.g., 906
Commer and Newman, 2009). 907
In some respects ModEM has been designed specically for frequency 908
domain EM geophysical problems e.g., the structure of data vectors re- 909
ects the transmitter/receiver structure common to such methods. However, 910
ModEM should be more broadly useful, in particular providing a good start- 911
ing point for development of joint inversion schemes. For example, ModEMs 912
data space structure already supports multiple data types, each of which may 913
require solution of a dierent forward problem e.g., as for the 2D MT prob- 914
lem discussed in EK12, where TE and TM mode data are inverted simultane- 915
ously. Joint inversion of EM and other sorts of geophysical data (e.g., seismic 916
or gravity) could be accommodated in a similar manner. In this case there 917
46
might be two separate sets of numerical discretization modules, with possibly 918
dierent grids, numerics, interpolation functionals, etc (see also Section 7). 919
The interface routines would then merge and encapsulate these, hiding the 920
details, and even the multiplicity of distinct physics involved. The two sorts 921
of data would of course have to be coupled through the model parameter, 922
either through some sort of constitutive relation, or through the covariance. 923
924
Of course, there will always be limits to the generality of any specic im- 925
plementation. For example, the generic inversion algorithms implemented 926
so far assume that the model regularization can be represented as a norm 927
on the model space, dened by a quadratic form C
m
. More general regular- 928
ization schemes would require some modication to these inversion modules, 929
although these could be implemented rather simply, making use of lower level 930
components. Modularity and code reuse in ModEM thus functions at mul- 931
tiple levels, from the numerical discretization to implementation of specic 932
inversion schemes. 933
934
9. Acknowledgements 935
The authors acknowledge support from the U.S. Department of Energy 936
grant DE-FG02-06ER15819, and National Science Foundation grant EAR1225496. 937
The authors thank Aihua Weng and Chatchai Vachiratienchai for helpful dis- 938
cussions of controlled-source EM inversion methods. 939
47
Appendix A. Modular Linearized Inversion Algorithms 940
A wide range of gradient based inversion algorithms can be implemented 941
using the components described above. As examples we summarize two 942
schemes we have implemented: non-linear conjugate gradients (NLCG, e.g., 943
Press et al., 1992), and the data-space conjugate gradient scheme (DCG) of 944
Siripunvaraporn and Egbert (2007). 945
946
For NLCG, we adopted the standard Polak-Ribi`ere scheme (see Figure 947
A.7 for pseudo-code). For the line search, which is based on Press et al. 948
(1992) and Nocedal and Wright (2006, Chapter 3), we rst interpolate using 949
a quadratic approximation; if the solution does not satisfy the sucient de- 950
crease (Armijo) condition (ignoring the curvature condition), we backtrack 951
using cubic interpolation. This strategy only requires one gradient evalua- 952
tion and is ecient when such computations are expensive. The initial step 953
size is set outside of the line search, and adjusted automatically based on 954
the rst gradient computation; it can also be specied by the user. We have 955
also implemented an automatic criterion to decrease the damping parameter 956
(, see EK12) when convergence of the inversion stalls. User control of the 957
inversion includes specifying the initial damping parameter, the sucient de- 958
crease condition, and the stopping criterion. 959
960
961
The DCG scheme of Siripunvaraporn and Egbert (2007) is an iterative 962
Gauss-Newton algorithm. Briey, the penalty functional is linearized about 963
a trial value of the model parameter m
n
, and the minimum of the linearized 964
48
m
prior
= prior model
= initial damping parameter
a = initial line search step scaled by model norm
n = 0
m
0
= 0
m
0
= C
1/2
m
m
0
+m
prior
= starting model
Evaluate P
0
= (d f(m
0
))
T
C
1
d
(d f(m
0
)) + m
T
0
m
0
Evaluate P

( m)|
m
0
= 2C
1/2
m
J
T
C
1
d
(d f(m
0
)) + 2 m
0
g
0
= P

( m)|
m
0
h
0
= g
0
Set
0
= a/g
T
0
g
0
While (d f(m
n
))
T
C
1
d
(d f(m
n
)) > T do
n = n + 1
Line search: choose to minimize P(C
1/2
m
( m
n1
+ h
n1
) +
m
prior
)
m
n
= m
n1
+ h
n1
P
n
= (d f(m
n
))
T
C
1
d
(d f(m
n
)) + m
T
n
m
n
g
n
= P

( m)|
mn
= 2(P
n
P
n1
)/g
T
n1
h
n1
Adjust for super-linear convergence: = 1.01
If too small
= 0.1
= a/g
T
n
g
n
Restart: h
n
= g
n
Cycle do loop
End if
= g
T
n
(g
n
g
n1
)/g
T
n1
g
n1
If (g
T
n
g
n
+ g
T
n
h
n1
> 0)
h
n
= g
n
+ h
n1
Else
Restart to restore orthogonality: h
n
= g
n
End if
End do
Figure A.7: Pseudo-code for non-linear conjugate gradient (NLCG) algorithm. Notation:
f(m) represents responses obtained from forward modeling; P
n
and P

n
are values of the
penalty functional and its derivative at nth iteration; m
n
, h
n
, g
n
are vectors in the model
space; and represent real scalars; other symbols are as in the text.
49
functional is sought. This leads to a system of normal equations involving 965
cross-products of J solved for b
n
with conjugate gradients (without actually 966
forming the normal equations). In the data-space variant we consider here, 967
these equations are 968
(JC
m
J
T
+ I)b
n
=

d
n
(A.1)
969
m
n+1
= C
m
J
T
b
n
. (A.2)
In the DCG algorithm Eq. (A.1) is solved for b
n
using conjugate gradients, 970
and the model update is computed as in (A.2). The whole procedure is 971
iterated over n to achieve the desired level of mist. Pseudo-code for this 972
scheme is provided in Figure A.8. Choice of the regularization parameter, 973
and stopping criteria for the conjugate gradient iterations are discussed in 974
Siripunvaraporn and Egbert (2007). For sensitivity computations we use 975
multiplication routines Jmult and JmultT, described in Table 2. 976
977
Akin, J. E., 2003. Object-oriented programming via Fortran 90/95, Volume 978
1317, Issue 1. Cambridge University Press. 979
URL http://books.google.com/books?hl=en&lr=&id=RZWprnAetU4C&pgis=1 980
Alumbaugh, D. L., Newman, G. A., Prevost, L., Shadid, J. N., 1996. Three- 981
dimensional wide band electromagnetic modeling on massively parallel 982
computers. Radio Science 33, 123. 983
Avdeev, D. B., Nov. 2005. Three-Dimensional Electromagnetic Modelling 984
and Inversion from Theory to Application. Surveys in Geophysics 26 (6), 985
767799. 986
50
m
prior
= prior model
m
0
= starting model
Outer loop: For n = 0, 1, 2, . . .

d
n
= d f(m
n
) +J[m
n
m
prior
]
Solve: [C
1/2
d
JC
m
J
T
C
1/2
d
+ I]b
n
=

d
n
using CG
x
0
= 0
r
0
= p
0
=

d
n
Inner loop (CG): For k = 1, 2, . . .
y
k
= C
1/2
d
JC
m
J
T
C
1/2
d
p
k
+ p
k

k
= r
T
k
r
k
/p
T
k
y
k
x
k+1
= x
k
+
k
p
k
r
k+1
= r
k

k
y
k
If ||r
k+1
|| < r
tol
b
n
= x
k+1
Exit
Else

k
= ||r
k+1
||/||r
k
||
p
k+1
= r
k+1

k
p
k
End if
End inner loop
m
n+1
= m
prior
+C
m
J
T
C
1/2
d
b
n
If (d f(m
n+1
))
T
C
1
d
(d f(m
n+1
)) < T exit
End outer loop
Figure A.8: Pseudo-code for data space conjugate gradient (DCG) algorithm, with normal
equations solved in the data space. Notation: f(m) represents responses obtained from
forward modeling;

d
n
, b
n
, x
k
, y
k
, p
k
and r
k
are all vectors in the data space;
k
and
k
represent real scalars; other symbols are as in the text.
51
Avdeev, D. B., Avdeeva, A., 2009. 3D magnetotelluric inversion using a 987
limited-memory quasi-Newton optimization. Geophysics 74 (3), F45F57. 988
Commer, M., Newman, G. A., Sep. 2009. Three-dimensional controlled- 989
source electromagnetic and magnetotelluric joint inversion. Geophysical 990
Journal International 178 (3), 13051316. 991
URL http://doi.wiley.com/10.1111/j.1365-246X.2009.04216.x 992
Constable, S. C., Parker, R. L., Constable, C. G., Mar. 1987. Occams in- 993
version: A practical algorithm for generating smooth models from electro- 994
magnetic sounding data. Geophysics 52 (3), 289300. 995
DeGroot-Hedlin, C., Constable, S. C., Jan. 2004. Inversion of magnetotel- 996
luric data for 2D structure with sharp resistivity contrasts. Geophysics 997
69 (1), 78. 998
URL http://geophysics.geoscienceworld.org/cgi/content/full/69/1/78 999
Egbert, G. D., 2012. Hybrid conjugate gradient-Occam algorithms for inver- 1000
sion of multi-frequency and multi-transmitter EM data. Geophys. J. Int. 1001
190, 255266. 1002
Egbert, G. D., Kelbert, A., 2012. Computational Recipes for Electromagnetic 1003
Inverse Problems. Geophys. J. Int. 189, 251167. 1004
Haber, E., Ascher, U. M., Oldenburg, D. W., 2004. Inversion of {3D} electro- 1005
magnetic data in frequency and time domain using an inexact all-at-once 1006
approach. Geophysics 69 (5), 12161228. 1007
Kelbert, A., Egbert, G. D., DeGroot-Hedlin, C., Mar. 2012. Crust and up- 1008
per mantle electrical conductivity beneath the Yellowstone Hotspot Track. 1009
52
Geology 40 (5), 447450. 1010
URL http://geology.gsapubs.org/cgi/doi/10.1130/G32655.1 1011
Kelbert, A., Egbert, G. D., Schultz, A., May 2008. Non-linear conjugate 1012
gradient inversion for global EM induction: resolution studies. Geophysical 1013
Journal International 173 (2), 365381. 1014
Key, K., 2009. 1D inversion of multicomponent, multifrequency marine 1015
CSEM data: Methodology and synthetic studies for resolving thin resistive 1016
layers. Geophysics 74 (2), F9-F20. 1017
McGillivray, P. R., Oldenburg, D. W., Ellis, R. G., Habashy, T. M., Jan. 1018
1994. Calculation of sensitivities for the frequency-domain electromagnetic 1019
problem. Geophysical Journal International 116 (1), 14. 1020
URL http://doi.wiley.com/10.1111/j.1365-246X.1994.tb02121.x 1021
Meqbel, N., Nov. 2009. The electrical conductivity structure of the Dead Sea 1022
Basin derived from 2D and 3D inversion of magnetotelluric data. Ph.D. 1023
thesis, Free University of Berlin, Berlin, Germany. 1024
Meqbel, N., Egbert, G. D., Wannamaker, P. E., Kelbert, A., Schultz, A., 1025
2013. Deep Conductive Structure of the Pacic NW derived from 3-D in- 1026
version of USArray Magnetotelluric data. Earth and Planetary Science 1027
Letters, accepted. 1028
Meqbel, N., Ritter, O., 2013. New Advances for a joint 3D inversion of multi- 1029
ple EM methods. In: Extended abstracts for the 5th International Sympo- 1030
sium on Three-Dimensional Electromagnetics, Sapporo, Japan, May 7-9, 1031
2013. 1032
53
Moorkamp, M., Heincke, B., Jegen, M., Roberts, A. W., Hobbs, R. W., 1033
Jan. 2011. A framework for 3-D joint inversion of MT, gravity and seismic 1034
refraction data. Geophysical Journal International 184 (1), 477493. 1035
URL http://doi.wiley.com/10.1111/j.1365-246X.2010.04856.x 1036
Newman, G. A., Alumbaugh, D. L., Feb. 1997. Three-dimensional massively 1037
parallel electromagnetic inversion I. Theory. Geophysical Journal Inter- 1038
national 128, 345354. 1039
Newman, G. A., Alumbaugh, D. L., 2000. Three-dimensional magnetotel- 1040
luric inversion using non-linear conjugate gradients. Geophysical Journal 1041
International 140, 410424. 1042
Newman, G. A., Boggs, P. T., Dec. 2004. Solution accelerators for large-scale 1043
three-dimensional electromagnetic inverse problems. Inverse Problems 20, 1044
151170. 1045
Newman, G. A., Hoversten, G. M., 2000. Solution strategies for two- and 1046
three-dimensional electromagnetic inverse problems. Inverse Problems 16, 1047
13571375. 1048
Nocedal, J., Wright, S. J., 2006. Numerical Optimization. Springer. 1049
Parker, R. L., 1994. Geophysical Inverse Theory. Princeton University Press, 1050
Princeton, New Jersey. 1051
Press, W. H., Teukolsky, S. A., Vetterling, W. T., Flannery, B. P., 1992. 1052
Numerical Recipes in Fortran 77 The Art of Scientic Computing, 2nd 1053
Edition. Cambridge University Press. 1054
54
Purser, R. J., Wu, W.-S., Parrish, D. F., Roberts, N. M., 2003a. Numerical 1055
Aspects of the Application of Recursive Filters to Variational Statistical 1056
Analysis. Part I: Spatially Homogeneous and Isotropic Gaussian Covari- 1057
ances. Monthly Weather Review 131 (8), 15241535. 1058
Purser, R. J., Wu, W.-S., Parrish, D. F., Roberts, N. M., 2003b. Numerical 1059
Aspects of the Application of Recursive Filters to Variational Statistical 1060
Analysis. Part II: Spatially Inhomogeneous and Anisotropic General Co- 1061
variances. Monthly Weather Review 131 (8), 15361548. 1062
Rodi, W. L., Mackie, R. L., 2001. Nonlinear conjugate gradients algorithm 1063
for 2-D magnetotelluric inversion. Geophysics 66 (1), 174187. 1064
URL http://link.aip.org/link/?GPY/66/174/1 1065
Siripunvaraporn, W., May 2011. Three-Dimensional Magnetotelluric In- 1066
version: An Introductory Guide for Developers and Users. Surveys in 1067
Geophysics 33 (1), 527. 1068
URL http://www.springerlink.com/index/10.1007/s10712-011-9122-6 1069
Siripunvaraporn, W., Egbert, G. D., Sep. 2007. Data space conjugate gradi- 1070
ent inversion for 2-D magnetotelluric data. Geophysical Journal Interna- 1071
tional 170, 986994. 1072
Siripunvaraporn, W., Egbert, G. D., Apr. 2009. WSINV3DMT: Vertical 1073
magnetic eld transfer function inversion and parallel implementation. 1074
Physics of the Earth and Planetary Interiors 173 (3-4), 317329. 1075
URL http://linkinghub.elsevier.com/retrieve/pii/S0031920109000144 1076
55
Siripunvaraporn, W., Egbert, G. D., Lenbury, Y., 2002. Numerical accuracy 1077
of magnetotelluric modeling: A comparison of nite dierence approxima- 1078
tions. Earth Planets and Space 54 (6), 721725. 1079
Siripunvaraporn, W., Egbert, G. D., Lenbury, Y., Uyeshima, M., 2005. 1080
Three-dimensional magnetotelluric inversion: data-space method. Physics 1081
of the Earth and Planetary Interiors 150 (1-3), 314. 1082
Smith, J. T., 1996. Conservative Modeling of 3-D Electromagnetic Fields: 1083
2. Biconjugate gradient solution and an accelerator. Geophysics 61 (5), 1084
13191324. 1085
Stefano, M. D., Andreasi, G. F., Re, S., Virgilio, M., Snyder, F. F., 2011. 1086
Multiple-domain, simultaneous joint inversion of geophysical data with 1087
application to subsalt imaging. Geophysics 76 (3), R69R80. 1088
Tarantola, A., 2005. Inverse Problem Theory and Methods for Model Param- 1089
eter Estimation. SIAM. 1090
Tietze, K., Ritter, O., Jul. 2013. Three-dimensional magnetotelluric inversion 1091
in practicethe electrical conductivity structure of the San Andreas Fault 1092
in Central California. Geophysical Journal International. 1093
URL http://gji.oxfordjournals.org/cgi/doi/10.1093/gji/ggt234 1094
Yee, K., May 1966. Numerical solution of inital boundary value problems 1095
involving Maxwells equations in isotropic media. IEEE Transactions on 1096
Antennas and Propagation 14, 302307. 1097
Zhdanov, M., Hursan, G., 2000. 3D electromagnetic inversion based on quasi- 1098
analytical approximation. Inverse Problems 16, 12971322. 1099
56
Zhdanov, M. S., 2002. Geophysical Inverse Theory and Regularization Prob- 1100
lems (Methods in Geochemistry and Geophysics). Elsevier Science. 1101
57