F3DT-FD Users Manual

Version 1.1

Po Chen , En_jui Lee & Xiaofeng Zhang September 3, 2010

CONTENT

Abstract ....................................................................................................................... 3 Chapter 1 Environment Setup ..................................................................................... 6 1.1 Log Onto The Cluster .................................................................................................................................... 6 1.2 Browse Work Path ............................................................................................................................................. 6 1.3 Compile Codes .................................................................................................................................................... 8 Chapter 2 Mesh Generation .................................................................................... 12 2.1 To generate seismic structure model ................................................................................................... 12 2.2 Generate Velocity Model .............................................................................................................................. 14 2.3 Source and Station models .......................................................................................................................... 17 Chapter 3 Generate HALFSPACE Model .................................................................. 21 3.1 What is the HALFSPACE model? ............................................................................................................... 21 3.2 EWF (earth-quack wave-field)Calculation .......................................................................................... 21 3.3 RGT(receiver strain green tensor) Calculation ................................................................................... 28 3.4 To generate synthetic seismograph ......................................................................................................... 30 3.5 Using Matlab software to check result ................................................................................................... 32 Chapter 4 Frchet Kernel ....................................................................................... 38 4.1 Kernel calculation on cluster ...................................................................................................................... 38 4.2 kernel draft on software ............................................................................................................................... 41

Abstract

In this manual, we will explain how to use the F3DT-FD package, which is a collection of tools for doing full-3D, full-wave seismic tomography and seismic source inversion based on the finitedifference method. Frequently Asked Questions: What is this software package really about? This package is not about traditional methods for imaging, it is about how to develop new and better methods for imaging. During this course, many classical methods for imaging or migration will be derived as special cases of a very general theory based on the wave equation, which we call the full-wave method, but the focus of this course is not about those special cases, but about this general theory itself. What do we mean by better imaging methods? There are two aspects about our meaning of better. On one aspect, better means more general, a more general method that can be applied to many different types of data gathered under very different circumstances. For example, in traditional imaging methods, we have the distinction between reflection and transmission. For transmission, we usually call our imaging methods tomography, for reflection, we usually call our imaging methods migration. The reason that we are having these distinctions is that some imaging methods work better for some types of data but do not work for other types of data. This course is about the unification of specialized methods under a general frame- work. A more general method has its advantages. First of all, as the amount of data available for analysis is exploding exponentially, it is getting more and more difficult to condition the data to meet the special needs of different imaging methods, so wed better come up with a more general method that can account for different types of data. Second of all, and more importantly, we would like to obtain an image or model that is consistent with different types of data and the best way to achieve this goal is to including those different types of data in a unified and self-consistent imaging method. A second aspect about being better is in terms of resolution of the obtained image. Strictly speaking, the imaging problem is usually ill-posed, in the sense that the observation gathered on a discrete grid is not sufficient to constrain a continuous model uniquely. At some point, we always need to in- corporate a-priori information into the imaging process in order to pick a model from

the large number of models that are consistent with the data. The a-priori information essentially extrapolates or interpolates the portion of the image that is constrained by the data to the portion that is not constrained by the data. By using a more general imaging method that can account for many different types of data, we can essentially incorporate more constraints to our model and minimize the use of a-priori information for model interpolation or extrapolation. Another important factor that limits the resolution in traditional imaging methods is the mathematical approximations, which, in essence, are adopted to reduce the cost for obtaining the solution. With the exponential growth in computing technology, in particular, distributed-memory parallel computing, our capability for obtaining solutions is dramatically improved compared with two
decades ago. As a result, many or all of the mathematical approximations can now be dropped and we can now employ an exact imaging method that does not sacrifice resolution for solvability.

Why is this software useful? From the practical perspective, the need for developing better imaging methods is motivated by the oil exploration industry. On one hand, the oil companies are making record profits from the high price of oil. On the other hand, they have to drill in increasingly complex geological environment under which traditional methods often fail to provide satisfactory images. In the past decade, oil companies have been investing more and more money into developing new imaging methods and the full-wave method is in the spotlight. Another very important application of full-wave imaging is in medicine. So far, X-ray tomography has been the working-horse in medical imaging. The underlying physics of X-ray tomography is essentially the so-called straight- ray tomography. The problem with X-ray tomography, in addition to the radiation introduced by the imaging process, is that it is primarily sensitive to density contrasts, but we need images of other types of properties in addition to density in order to reach reliable diagnoses. This problem is more evident in cancer imaging, in which case, the density of the tumor is not too different from ordinary tissue. In the past decade, people have been building new instruments for cancer imaging using acoustic waves, which can provide images of acoustic wave speed, attenuation property in addition to density. Accurate imaging of acoustic scattering in human body will require a full-wave approach. 3D Seismic Wave-Propagation Simulations Based on the Finite-Difference (FD) Method The finite-difference code for solving the elastic/anelastic seismic wave equation is name pmcl3d. This code can handle 3D seismic velocity models (specified by density , P-wave speed Vp and S-wave speed Vs as functions of position), 3D attenuation models (Qp and Qs specified as functions of position) and seismic sources with finite size.

So far pmcl3d does not handle anisotropy, does not account for topography or curvature of the Earth. It can only use a mesh with uniform grid size which does not vary with respect to position or direction. Synthetic strain field is very useful for constructing sensitivity (Frchet) kernels for tomography because the calculation of the kernel involves a temporal convolution of the strain fields from the seismic source and from the receiver at every spatial grid point. To help users prepare the input files in the format that pmcl3d reads and to convert the outputs of pmcl3d into formats that are more readable by human or another program, we have developed a set of Fortran codes and python or shell scripts. In this software package the only thing what the user only need to do is execute these Fortran and Python code to generate sensitivity kernels.

Chapter 1 Environment Setup

1.1

Log Onto The Cluster

For any user who wants to run our codes, the first thing to do is to log onto the cluster where our codes stored. We can login as follows, ssh class8@seismicmst.gg.uwyo.edu Here, ssh means remote login. After longin, computer screen will ask you about password, as follow, class8@seismicmst.gg.uwyo.edus password: Then you type the password classeyGG . If you type the password correctly it will show you your last time login information. For example: Last login: Tue Jan 12 13:01:13 2010 from feng-pc uwyo.edu

1.2 Browse Work Path

Since we have already login, let me show what does the work path look like. [class8@seismicmst ~] $ cd /seismic04/class8 By doing commend ls you will see 3 directories as bellow, [class8@seismicmst class8] $ ls
F3DT HALFSPACE test_MPI

Let us go forward to see what do it looks like in these directories, [class8@seismicmst class8] $ cd F3DT [class8@seismicmst F3DT] $ ls
F3DT_FD_1.1

[class8@seismicmst F3DT] $ cd F3DT_FD_1.1 [class8@seismicmst F3DT_FD_1.1] $ ls

bin scripts src templates

Let us go forward to diretory bin .All the executable files store here, showing as below, [class8@seismicmst F3DT_FD_1.1] $ cd bin [class8@seismicmst bin] $ ls

asc2bin ewf2syn FD_GRID_XYZ_rect

ker_collect_bin ker_filter_par latlon2grd

pmcl3d rgt2syn rotate_syn

update_structure write_source

Directory scripts stores the python scripts files, showing as below,

[class8@seismicmst F3DT_FD_1.1] $ cd ../scripts [class8@seismicmst scripts] $ ls

evelist_gen.py ewf2syn_rotate3comp.py set_source_location.py ewf2syn_rotate.py F3DT_Init.py F3DT_Init.pyc gen_jobs.py latlon2grd.py pmcl3d_setup.py pre_run.py rgt2syn_rotate.py run_gen_jobs.sh set_receiver_location.py set_sgt_location.py stalist_gen.py write_IN3D.py write_source.py

Directory src store source information such like forward model, inverse model and measure tools, showing as below, [class8@seismicmst F3DT_FD_1.1] $ cd ../src [class8@seismicmst src] $ ls
forward inverse measure

Directory templates stores templates files and example files, showing as below, [class8@seismicmst F3DT_FD_1.1] $ cd ../.templates [class8@seismicmst templates] $ ls
SETTING src stf_grn_3Hz.dat stf_grn_5Hz.dat stf_grn_5s.dat update_structure.inp

Before we show the HALFSPACE directory one thing we need to know first is what directory HALFSPACE is. Directory HALFSPACE is the work folder where we generate halfspace model. That means this folder/directory should be empty, we create the input files, run codes and get the output files under this directory. In the practice or real task we create a directory named HALFSPACE and we do all our jobs under it. We will discuss the details later. Now let us see what should HALFSPACE directory look like. Since we are now in the directory F3DT/F3DT_FD_1.1/templates we need to go back 3 directories to get target directory HALFSPACE. The commends show as bellow, [class8@seismicmst templates] $ cd ../../../HALFSPACE [class8@seismicmst HALFSPACE ]$ ls
EWF KER JOBS RGT SYN VM

These six directories are what we need to create in next step. And all our jobs will be done in these directories.

1.3 Compile Codes

For every new user, before we start to run these codes you need to download the directory F3DT/F3DT_FD_1.1 to your own class. Every new user is supposed to occupy a class like class8. Since every user may have different environment like different Linux System, Python edition, Fortran edition, C++ program edition, another necessary thing except login you need to do is compiling these codes. This step will ensure these codes match your PC system. That means after you download directory F3DT/F3DT_FD_1.1 from the cluster, you need to clean bin (bin directory is the place we store executable files) directory in order to get the newest executable files. Do not bet these codes will work on your own PC without compile them. And you have to do this no matter what cluster you are on. For example, I am now on the cluster of seismicmst , we compile these codes as follows, [class8@seismicmst ~] $ cd /seismic04/class8/F3DT/F3DT_FD_1.1/src/forward [class8@seismicmst forward] $ ls
bound.f structure.f bound.o structure.o cerjan.f cerjan.o custompi.f swap_sync.f custompi.o io.f viscoop.f io.o viscoop.o makefile makefile.abe makefile.alpha makefile.bgl makefile.ia64 makefile.intrepid makefile.lonestar makefile.old makefile.pwr4 makefile.seismicmst makefile.usc makefile.xt3 md5.c md5.h md5.o memory.f memory.o operator.f operator.o parstat.mod pmcl3d.f pmcl3d.f~ pmcl3d.f.bak pmcl3d.o pmd5.c pml.f pml.o relax.f relax.o set_names.f set_names.h set_names.o sgsndyna.f sgsndyna.o sgsnswap.f sgsnswap.o source.f source.o station.f station.o swap.f swap.o test

[class8@seismicmst forward] $ makefile f makefile.seismicmst For example, If you are on the cluster of ia64 what we need to do is to make f makefile.ia64 . Now let us look forward about what makefile.seismicmst looks like. There are two makefile.seismicmst files separately exist in directory forward and directory inverse. Showing forward/makefile.seismicmst as an example, [class8@seismicmst forward] $ more makefile.seismicmst
OFLAGS = mpif90 O3 -c -xT -mp1 -extend_source -o PFLAGS = mpif90 -O3 -choicemod -xT -mp1 -v -extend_source -o BIN = ../../bin OBJECTS = memory.o bound.o relax.o swap.o io.o source.o \ custompi.o station.o structure.o operator.o \ viscoop.o pml.o cerjan.o pmcl3d.o md5.o set_names.o \ sgsndyna.o sgsnswap.o pmcl3d: $(OBJECTS)

$(PFLAGS) pmcl3d mv pmcl3d $(BIN)

$(OBJECTS)

$(LIB)

md5.o: md5.c mpicc -O2 -c -g -o $@ md5.c pmcl3d.o: pmcl3d.f $(OFLAGS) pmcl3d.o sgsndyna.o: sgsndyna.f $(OFLAGS) sgsndyna.o ...... pmcl3d.f sgsndyna.f

Let me interpret this file. 1st line OFLAGS = mpif90 O3 -c -xT -mp1 -extend_source o is a compile command. OFLAGS represents executing commend mpif90 O3 -c -xT -mp1 -extend_source o mpif90 specify open mpif90 compiler. O3 specify optimize for maximum speed and enable more aggressive optimization -c specify compile to object (.o) only , do not link

-xT specify direct links to read link commands form file -mp1 specify improve floating point precision -extend_source specify rightmost column for fixed from source o specify name output file

2nd line PFLAGS = mpif90 -O3 -choicemod -xT -mp1 -v -extend_source -o is also a compile command. -choicemod -v specify displace compiler information

3rd line BIN = ../../bin is a path. BIN specify path ../../bin 4th line OBJECTS = memory.o bound.o relax.o swap.o io.o source.o \ custompi.o station.o structure.o operator.o \ viscoop.o pml.o cerjan.o pmcl3d.o md5.o set_names.o \ sgsndyna.o sgsnswap.o are files.

OBJECTS represents object files memory.o sgsnswap.o 5th line pmcl3d: $(PFLAGS) pmcl3d $(OBJECTS) $(OBJECTS)

mv pmcl3d $(BIN) pmcl3d: $(OBJECTS) represents to do pmcl3d need to use OBJECTS. The OBJECTS is files memory.o sgsnswap.o that I have specified in the file. $(PFLAGS) pmcl3d $(OBJECTS) represents this PFLAGS commands output file is pmcl3d, and this compile command will use OBJECTS binary files. mv pmcl3d $(BIN) represents move pmcl3d to path BIN. 6th line md5.o: md5.c mpicc -O2 -c -g -o $@ md5.c represents to get binary file md5.o need to use file md5.c, and the compile command is
mpicc -O2 -c -g -o $@ md5.c

The following lines have the same meaning as line 6th. After compile command I showed in last step, [class8@seismicmst forward] $ makefile f makefile.seismicmst We get pmcl3d executable file in the directory of bin, showing as follows, [class8@seismicmst forward] $ cd ../bin [class8@seismicmst bin] $ ls
pmcl3d

The other executable files in the bin directory. You can get them through the same compiling command as we used in the forward directory, showing as below, [class8@seismicmst bin] $ cd ../src/inverse [class8@seismicmst inverse] $ ls
asc2bin.cpp box2en_rot.f90 box2rt.f90 chebyshev_filter_sub.f90 chebyshev_filter_sub.o en2box_rot.f90 ewf2syn.f90 #FD_GRID_XYZ_rect.f90# FD_GRID_XYZ_rect.f90 FD_GRID_XYZ_rect.f90~ findnode.f90 grd2utm.f90 ker_collect_bin.f90 ker_filter_par_v3.f90 ker_filter_par_v3.o ker_sub.f90 ker_sub.mod ker_sub.o latlon2grd.f90 makefile makefile~ makefile.seismicmst makefile.seismicmst~ rgt2syn.f90 rgt2syn.f90~ rgt2syn.f90_bak rotate_syn.f90 update_structure.f90 update_structure.f90~ utm2grd.f90 utm_geo.f90 write_source.f90

[class8@seismicmst inverse] $ makefile f makefile.seismicmst [class8@seismicmst inverse] $ cd ../../bin [class8@seismicmst bin] $ ls
asc2bin ewf2syn FD_GRID_XYZ_rect ker_collect_bin ker_filter_par latlon2grd pmcl3d rgt2syn rotate_syn update_structure write_source

10


So far we have already introduced how to login cluster, how are our codes directories organized and compiled. The coming next 4 chapters will be our main parts of this user menu. They will show you how to run these codes and how to generate kernel step by step.

11

Chapter 2

Mesh Generation

2.1 To generate seismic structure model

To generate mesh is always the first step before we do any geologic simulation. Here is our example to built geologic grid. Step 1 To prepare work directory. Following the format of F3DT software package we need to create HALFSPACE directory and submit. To avoid we have the same with the existent HALFSPACE directory we create a directory named HALFSPACE2, commend showing as below, [class8@seismicmst HALFSPACE ]$ cd ../ [class8@seismicmst HALFSPACE ]$ ls
F3DT HALFSPACE test_MPI

[class8@seismicmst class8 ]$ mkdir HALFSPACE2 [class8@seismicmst class8 ]$ ls

F3DT HALFSPACE test_MPI HALFSPACE2

[class8@seismicmst class8 ]$ cd HALFSPACE2 [class8@seismicmst HALFSPACE2 ]$ mkdir EWF KER JOBS RGT SYN VM [class8@seismicmst HALFSPACE2 ]$ ls
EWF KER JOBS RGT SYN VM

As I mentioned before, directory JOBS is the work place. Our next work will be done in this directory. Step 2 To prepare input files. As the codes requires, the input files should named as FD_RGID_XYZ_HS.inp. In this file, it should specify spatial volume of mesh and grid space basic on the realistic target, it should also specify processors distribution in order to make each processor computes same or approximate numbers of grids which can guarantee computational efficiency. The format of the input file shows as bellow, [class8@seismicmst HALFSPACE2 ]$ cd JOBS [class8@seismicmst JOBS] $ emacs FD_RGID_XYZ_HS.inp

12

3 N
/seismic04/class8/HALFSPACE/VM 4 112 -118.0 33.0 -118.125 33.0 47500 35500 1 200 E 8 2 6

Fig.2.1.1 Box location corresponding geographic orientation

Within the input file, /seismic04/class8/HALFSPACE/VM 11 -118.0 -118.125 47500 35500 200 8 2 6 33.0 33.0

# the directory for the output grid file XYZGRD # the UTM zone of the geographic box # longitude and latitude of point 1 # longitude and latitude of point 2 # length of the box in meters # specifies the thickness of the modeling volume in meters # the grid spacing in meters

# the number of processors assigned to each dimension of the mesh

For the longitude values, a negative value represents west, a positive value represents east. The longitude and latitude values should be specified as floating-point numbers.

The Fig.2.1.1 is the geographic box. The lower left point is point 1; the upper left point is point 2. If we count the four corners clockwise it starts from point 1. The box length is specified by the distance between point 1 and point 2 that should be 47500 meters. Step 3 To generate structured geographic model (mesh). Do you remember that we have already compiled some codes in chapter 1? One of them named FD_GRID_XYZ_rect will be use here to generate mesh. Since we have prepared the input file FD_RGID_XYZ_HS.inp, we can execute the executable and generate mesh now, showing as below,

13

[class8@seismicmst JOBS]$ FD-GRID-XYZ_rect FD_RGID_XYZ_HS.inp If the code is executed without error, it will generate an output file XYZGRD in the directory VM and box.dat in directory JOBS. Details show as below, [class8@seismicmst JOBS]$ ls
box.dat FD_RGID_XYZ_HS.inp

[class8@seismicmst JOBS] $ emacs box.dat

-118.000000000000
-118.019000000000 -118.019487698072 -118.000007364373 -118.000000000000

33.0000000000000 33.0000000000000 33.0494074502728 33.0494114938546 33.0000000000000

These six lines numbers specify point 1, 2, 3, 4, 1 on geographic box.

[class8@seismicmst JOBS]$ cd../VM

[class8@seismicmst VM]$ ls
XYZGRD

[class8@seismicmst VM]$ emacs XYZGRD

240 8 30 60 180 ! Global grid numbers NX, NY, NZ 2 6 ! Chunks (#of proc.) in X, Y, Z 30 30 ! Grid numbers per chunk NXT, NYT, NZT 406580.173921413 3651527.06305781 394780.775856964 3651645.02359504 395258.615999296 3699442.58524595 407058.014063745 3699324.62470872 180 0.000000000000000E+000 179 -200.000000000000 178 -400.000000000000 177 -600.000000000000 176 -800.000000000000

.
1 -35800. 000000000000

As we see from the output file, the geographic box is divided to 240*60*180 grids and each processor deals with 30 grids which accord with our processor distribution policy. In the vertical, the grids count form bottom to up. The first layer of grids is on bottom of mesh. The last layer of grids is on the on the top of mesh that is the surface of ground. We can check the output file from the third line that is the number of girds at each processor on each direction, if the three numbers close to each other, the processors we assigned to each directory is correct.

2.2 Generate Velocity Model

Velocity model is the model that specified density, Vp and Vs on each grid. Here, we also coded C++ codes named gen_half_mdl.cpp for generating this velocity model. But we didnt write it into makefile that we compiled in chapter one because different user may want generate different velocity model such like different properties in different grid. So it is convenience for

14

users code their own velocity C/C++ program. The program gen_half_mdl.cpp we have coded specifies each grid has same properties, showing as bellow, [class8@seismicmst VM]$ cd../JOBS [class8@seismicmst JOBS]$ cp../HALFSPACE2/JOBS/gen_half_mdl.cpp ./ [class8@seismicmst JOBS]$ ls
box.dat FD_RGID_XYZ_HS.inp gen_half_mdl.cpp

[class8@seismicmst JOBS]$ more gen_half_mdl.cpp
int main() { // setting processors in X, Y, and Z const int NPX=8; const int NPY=2; const int NPZ=6; // number of grids in dimension of X, Y, Z const int NX=240; const int NY=60; const int NZ=180; // setting Vp (m/sec), Vs (m/sec), and density :rho (kg/m^3) const float vp=6800.0; const float vs=4000.0; const float rho=3000.0; // number of grids in each processor int NXT=NX/NPX; int NYT=NY/NPY; int NZT=NZ/NPZ; char mdl_out[20]={0}; //loop for generate velocity model files for (int ipz=0; ipz<NPZ; ipz++){ for (int ipy=0; ipy<NPY; ipy++){ for (int ipx=0; ipx<NPX; ipx++){ // generate file name sprintf(mdl_out,"HSGRD%3.3d%3.3d%3.3d",ipx,ipy,ipz); // open outfile FILE *bin_out; bin_out=fopen(mdl_out,"wb"); for (int z_no=-2; z_no<=NZT+1; z_no++){ for (int y_no=-2; y_no<=NYT+1; y_no++){ for (int x_no=-2; x_no<=NXT+1; x_no++){ fwrite (&vp,sizeof(float),1,bin_out); fwrite (&vs,sizeof(float),1,bin_out); fwrite (&rho,sizeof(float),1,bin_out); } } } fclose(bin_out); } } }

15

return 0; }

Let us do a little analysis of this C++ codes. Symbol // represents the rest of line is notation that do not participate in the computing, only provide users whats the function of rest lines of codes. As it explained in the notation, the function of this program first specifies the number of processors and the number of grids calculated on each processor in each dimension. Then, it specifies the constant value of density, Vp and Vs. Last, it assigns these values in each grid through three loop nets. In the first loop it assigns the calculation on each processor. In the second loop it assigns properties on each grid in one processor. This file names a 9-digit number that specifies the processor index. The option 'l' in the last three lines specifies that the output format should be binary in \little-endian". Line 29 closes the seismic structure file for each processor. This Matlab script will generate a set of binary files named such as HSGRD001001001 that means this file stores the information of the first processor in dimension X, Y, Z. You can modify this Matlab script to generate much more complex 3D seismic structure models by specifying the P- and S-wave speeds and density as functions of global grid number, which is computed on lines 21-23. Before we execute this program we still need to compile it. As we mentioned in chapter 1 that can make sure the codes edition match with your environments. Commands show as bellow, [class8@seismicmst JOBS]$ icc o gen_half_mdl gen_half_mdl.cpp
icc o gen_half_mdl gen_half_mdl.cpp

# compile C/C++ program to executable # file # executable # input C/C++ program

[class8@seismicmst JOBS]$ gen_half_mdl [class8@seismicmst JOBS]$ ls

XYZGRD HSGRD001001001

Because we did not specifies the directory of output file, so the output files store where you execute the command. For convenient you can either copy these output files to directory VM or execute gen_half_mdl on directory VM. [class8@seismicmst JOBS]$ mv HSGRD* ../VM [class8@seismicmst JOBS]$ cd ../VM When we analysis program gen_half_mdl.cpp we acquire the conclusion that the output files will store properties of each grid in each processor. Now we can check our output files simply like this, if the number of output files equal to the numbers of processor we used in computing, the output files are correct. Commands show as follows, [class8@seismicmst JOBS]$ ls HSGRD00* | wc l 96

16

2.3 Source and Station models

Source and station models present the source or receivers relative position in a mesh. In the rest of the section we will use an important parameter file named param_HS.dat to participate in execution. Step 1 To prepare parameter file. [class8@seismicmst JOBS]$ emacs param_HS.dat
BINDIR = /seismic04/class8/F3DT/F3DT_FD_1.1/bin SCRIPTDIR = /seismic04/class8/F3DT/F3DT_FD_1.1/scripts EWFDIR = /seismic04/class8/HALFSPACE2/EWF RGTDIR = /seismic04/class8/HALFSPACE2/RGT KERDIR = /seismic04/class8/HALFSPACE2/KER INVDIR = /seismic04/class8/HALFSPACE2/INV JOBDIR = /seismic04/class8/HALFSPACE2/JOBS DATDIR = /seismic04/class8/HALFSPACE2/DATA SYNDIR = /seismic04/class8/HALFSPACE2/SYN XYZGRD = /seismic04/class8/HALFSPACE2/VM/XYZGRD UTMZONE = 11 MEDIA = /seismic04/class8/HALFSPACE2/VM/HSGRD MINIMUM VELOCITY = 4000.0 MAXIMUM VELOCITY = 6880.0 SETTING = /seismic04/class8/F3DT/F3DT_FD_1.1/templates/SETTING SOURCE TIME FUNCTION = /seismic04/class8/F3DT/F3DT_FD_1.1/templates/src GREEN TENSOR FILTER = /seismic04/class8/F3DT/F3DT_FD_1.1/templates/stf_grn_3Hz.dat SOURCE LIST = /seismic04/class8/HALFSPACE2/JOBS/eveinfo/evelist.foc RECEIVER LIST = /seismic04/class8/HALFSPACE2/JOBS/stainfo/stalist.foc SOURCE STENCIL SIZE IN X = 1 SOURCE STENCIL SIZE IN Y = 1 SOURCE STENCIL SIZE IN Z = 1 TMAX = 30.0 NTSKP = 2 NXSKP = 2 NYSKP = 2 NZSKP = 2 CFL = 0.9792 USEPML = 1 PMLCOEF = 3.5 BOUNDARY NODE NUMBER = 10 VISCO = 1 SOURCE GRID POINTS = 1

In this file, if the behind of the equation is path then the capital letter before the equation represents the short for the path. For example, path /seismic04/class8/F3DT/F3DT_FD_1.1/bin will be substituted by BINDIR in codes which embrace this parameter file. UTMZONE: There are 60 longitudinal projection zones numbered 1 to 60 starting at 180W. Each of these zones is 6 degrees wide. There are 20 latitudinal zones spanning the latitudes 80S

17

to 84N and denoted by the letters C to X. Each of these is 8 degrees south-north, apart from zone X which is 12 degrees south-north.
SOURCE STENCIL SIZE IN X = 1

# sources relative position in mesh extend one grid at

each hand in direction of X

SOURCE STENCIL SIZE IN SOURCE STENCIL SIZE IN TMAX = 30.0 NTSKP = 2 NXSKP = 2 NYSKP = 2 NZSKP = 2 CFL = 0.9792 USEPML = PMLCOEF = BOUNDARY NODE NUMBER = VISCO = SOURCE GRID POINTS = Y = 1 Z = 1

# the record time for this explosion is 30 seconds # skip every other time step # skip every other grid in direction X . # adjustable parameter affect computing speed
1 3.5 10 1 1

Step 2 To prepare receiver and station input files. We put these two input files in two subdirectories, stainfo and eveinfo. [class8@seismicmst JOBS]$ mkdir stainfo eveinfo [class8@seismicmst JOBS]$ cd stainfo [class8@seismicmst stainfo]$ emacs stalist.foc
1 1 1 1 1 1 30 30 60 IN RC01 30 30 180 SF RC02 210 30 180 SF RC03

column1 1 present receiver, source is represented by 0 column2 1 present grid coordinate, geographic coordinate is represented by 0 column1 30 present receivers location in dimension X at grid coordinate column1 30 present receivers location in dimension X at grid coordinate column1 60 present receivers location in dimension X at grid coordinate column1 IN present receiver underground, receiver on surface is presented by SF column1 RC01 present receivers ID [class8@seismicmst JOBS]$ cd ../eveinfo [class8@seismicmst JOBS]$ emacs evelist.foc
0 1 1 1 1 1 210 30 60 1 1 1 1 0 0 0 100001 30 30 180 SF RC02 210 30 180 SF RC03

Within this file, the first number in the first line is 1 that present the first line is sources information and the first number at line 2 and line 3 is 0 that present these two lines is receiver information. Here, we consider the two receivers as source too. The receiver information has the same format as stalist.foc. About the source information the first 5 columns represent same meaning as stalist.foc. In the first line, the sixth number 1 presents moment tensor. The following six numbers present explosive type of source. The last number is the ID of source. Step 3 To generate receiver model, source model and source green function.

18

The receiver model here means there are some grids in the structured mesh record receivers location. You can interpret the source model and source green function following the same logical. Here is an example to generate receiver model, [class8@seismicmst JOBS]$ /seismic04/class8/F3DT/F3DT_FD_1.1/scripts/set_receiver_location.py ./param_HS.dat [class8@seismicmst JOBS]$ cd ../VM [class8@seismicmst JOBS]$ ls
XYZGRD HSGRD001001001 . HSGRDRECSTA001001000

[class8@seismicmst JOBS]$ ls HSGRDRECSTA00* | wc l 96 From the analysis of param_HS.dat we can see that SOURCE STENCIL SIZE is setting up to 1 in three dimensions. That means when we calculate source model the mesh will record (1+2) grids in each dimension. For each source the model will record 27 grids. Here is the example that we generate source gird model, [class8@seismicmst JOBS]$ /seismic04/class8/F3DT/F3DT_FD_1.1/scripts/set_source_location.py ./eveinfo/evelist.foc [class8@seismicmst JOBS]$ cd ../VM [class8@seismicmst JOBS]$ ls
XYZGRD HSGRD001001001 . HSGRDRECSTA001001000 HSGRDRECSRC004001002

./param_HS.dat

[class8@seismicmst JOBS]$ ls HSGRDRECSRC00* | wc l 96 To generate source green function model, [class8@seismicmst JOBS]$ /seismic04/class8/F3DT/F3DT_FD_1.1/scripts/set_sgt_location.py ./param_HS.dat [class8@seismicmst JOBS]$ cd ../VM [class8@seismicmst JOBS]$ ls
XYZGRD HSGRD001001001 .

19

HSGRDRECSTA001001000 HSGRDRECSRC004001002 HSGRDREC001001001 .

[class8@seismicmst JOBS]$ ls HSGRDRECS00* | wc l 96 This source green function model is to generate the files that specify the grid points at which we want to store the computed stain field, we call these grid points \recording grids". The recording grids should be a subset of the computing grids. This step is for computing Frechet kernels when we do tomography. It is optional if we only want to compute synthetic seismograms at a set of receiver locations.

20

Chapter 3

Generate HALFSPACE Model

3.1 What is the HALFSPACE model?

In geometry, a half-space is either of the two parts into which a plane divides the threedimensional space. Here we use Halfspace model to divide our three dimension world to air and one layer, and it is isotropic in the layer.

3.2 EWF (earth-quack wave-field)Calculation

EWF calculation is forward calculation that means calculate wave-form source to station. We have already write this step to a python code named pmcl3d which we also have already compiled in chapter 1. Here what we need to do is to execute this code. To execute this code we need some input. The fist one is param_HS.dat file. This input file tells pmcl3d code the basic parameter like velocity, source time function and so on (details see param_HS.dat file in chapter 2). The second input is forward or inverse simulation that you want to do. If you want to do forward simulation then type EWF at second input location, after you excute the code it will generate job files at the derectory that EWF represent at param_HS.dat file. Oppsitelly, if you want to do inverse calculation then type RGT , samely, it will generate job files at the direcoty that RGT represent at param_HS.dat file. The third input is a number, this number represent the beginning shot number you want to calculate and the fourth input is also a number, this number represnet the end shot number that you want coculate. Now, let me show an example: [class8@seismicmstm JOBS]$ /seismic04/class8/F3DT/F3DT_FD_1.1/scripts/pmcl3d_setup.py param_HS.dat EWF 1 3 In this example, the command represents to do forward computing from shot 1 to shot 3. It is nesseray to give the full directory of where the pmcl3d code is. To be convenience we can write this command to a script. We have to name the script as *.sh or *.csh that because it can be changed to executable. Steps is showing as below, [class8@seismicmstm JOBS]$ emacs pmcl3d_EWF.csh

21

Then write the command line showing before in to the file. To execute the file is just one command, showing as follow, [class8@seismicmstm JOBS]$ chmod +x pmcl3d_EWF.csh Then you will see the pmcl3d_EWF.csh file become green that mean executable here. We also have another way to execute the *.sh or *.csh file except using command to execute it. Using sh before *.sh or *.csh file can get the same result s, just like sh pmcl3d_EWF.csh . To see the jobs files generated by this command we need to go to the directory that EWF present. [class8@seismicmstm JOBS]$ cd ../EWF [class8@seismicmstm JOBS]$ pwd [class8@seismicmstm JOBS]$ /seismic04/class8/HALFSPACE/EWF [class8@seismicmstm EWF]$ ls 100001 100002 100003 [class8@seismicmstm EWF]$ cd 100001/ [class8@seismicmstm 100001]$ ls
100001.src 100001.src.inp input_rst output_ckp output_sfc pmcl3d IN3D output_dyn output_vlm SETTING

All this files are generated by last command. The 100001.scr file is the souce time function file. It has 151 lines and the fist line is recored the source location as X, Y, Z direction. The other 150 lines is the source time function. You can down load the this file and draw the first column to see what dose this source time function looks like. [class8@seismicmstm 100001]$ head 100001.src
210 30 0.1176430E+15 0.0000000E+00 0.1761050E+15 0.0000000E+00 0.2620290E+15 0.0000000E+00 0.3875250E+15 0.0000000E+00 0.5696690E+15 0.0000000E+00 121 0.1176430E+15 0.1748786E+02 0.1761050E+15 0.2617835E+02 0.2620290E+15 0.3895111E+02 0.3875250E+15 0.5760634E+02 0.5696690E+15 0.8468240E+02 0.1176430E+15 0.1761050E+15 0.2620290E+15 0.3875250E+15 0.5696690E+15 0.0000000E+00 0.0000000E+00 0.0000000E+00 0.0000000E+00 0.0000000E+00

[class8@seismicmstm 100001]$ wc 100001.src [class8@seismicmstm 100001]$ 151

22

File 100001.src.inp is the input file that used to generate 100001.src. You do not need to care about this file as a user. It is generated by the python code and called by the python code. [class8@seismicmstm 100001]$ more 100001.src.inp
/seismic04/class8/HALFSPACE/VM/XYZGRD -1 /seismic04/class8/HALFSPACE/EWF/100001/100001.src 1 210 30 60 0 0 0 1 1 1 0 0 0 0.0142325581395 -99 -99 /seismic04/class8/F3DT/F3DT_FD_1.1/templates/src 150

The pmcled file is an executable. [class8@seismicmstm 100001]$ ls al /pmcl3d [class8@seismicmstm 100001]$ /seismic04/class8/F3DT/F3DT_FD_1.1/bin/pmcl3d IN3D file is an important input file during this calculation like file 100001.src.inp, it is generated and called . directory output_* are the output derectory. The directory output_ckp is where store the check files. [class8@seismicmstm 100001]$ more IN3D -1 --earthquack source 30.0 --maxim calculating time 200.0 -- grid space 0.0142325581395 --sampling interval 1 10 3.5 1 --source number 150 -- time step 240 --grid number on X direction 60 --grid number on Y direction 180 --grid number on Z direction 8 --processer number on X direction 2 --processer number on Y direction 6 --processer number on Z direction 1 --skip number 240 1 1 60 1 1 1 1 1000000 1

23

240 2 1 60 2 1 180 2 2 1 0.677 0.525 0.005 2.0 1.0 'output_ckp/CHKP96PV' --output check file 'output_ckp/CHKJ96PV' --output check file '/seismic04/class8/HALFSPACE1/EWF1/100001/100001.src' --input file '/seismic04/class8/HALFSPACE1/VM1/HSGRD' --input file 'output_sfc/SX96PS' --output file 'output_sfc/SY96PS' --output file 'output_sfc/SZ96PS'- --output file 'output_vlm/SX96PV' --output wave field 'output_vlm/SY96PV' --output wave field 'output_vlm/SZ96PV' --output wave field 150 --time step 1 1 Directory output_ckp is the check directory. File CHKJ96PV is the checking file for all the parameters like velocity, density, time step and so on. These parameters should math the original value. The one I want emphasize is on the second line about STABILITY CRITERIA in the second line, this number should less than 0.5, it will be better if it close to 0.5. [class8@seismicmstm 100001]$ cd output_ckp/ [class8@seismicmstm output_ckp]$ ls CHKJ96PV CHKP96PV [class8@seismicmstm output_ckp]$ more CHKP96PV

igreen =

-1 --earth-quack source STABILITY CRITERIA .5 > CMAX*DT/DX = 0.4839070E+00 stability parameter<0.5 -# OF X NODES PER PROC 30 # OF Y NODES PER PROC 30 # OF Z NODES PER PROC 30 # OF TIME STEPS 2109 DISCRETIZATION IN SPACE 200.0000 DISCRETIZATION IN TIME 0.0142

--

24

PML REFLECTION COEFFICIENT 3.500 HIGHEST P-VELOCITY ENCOUNTERED 6800.00 ......

Directory output_vlm is wave field output directory. Now it is empty. It will generate wave filed after we submit the job. So far directory input_rst, output_syn, output_sfc have not been used. After we introduce these job files system, the next step is submit this job. We can submit thses job seperately or together with in a script. But before that we need to creat another file named mymachines In this file it is recored the processors name. Here I already edit it, so I just need to copy it to the current directory. [class8@seismicmstm JOBS]$ cd /seismic04/class8/HALFSPACE/EWF/100001 [class8@seismicmstm EWF]$ ls
100001.src 100001.src.inp input_rst output_ckp output_sfc pmcl3d IN3D output_dyn output_vlm SETTING

[class8@seismicmstm EWF]$ cp ../../../mymachines ./ [class8@seismicmstm EWF]$ more mymachines

seismic01m seismic02m seismic31m seismic32m seismic01m seismic02m seismic31m seismic32m seismic01m seismic02m seismic31m seismic32m seismic01m seismic02m seismic31m seismic32m

You can see from the file mymachines that there are 32 nodes in this cluster from seismic01m to seismic32m and each node is made up by 4 processors. There is some advantage for us to use this file. The biggest one is if there is some nodes do not work that wont affect the whole

25

cluster, the simple operation is just to delete the disable nodes from file mymachines. Actually, there is no need to put file mymachines in this directory. You can put it anywhere you want, just to be sure given the right directory when you are calling it. There are some examples for you to submit the job: [class8@seismicmstm EWF]$ cd 100001 [class8@seismicmstm 100001]$ mpirun_rsh -hostfile mymachines np 96 pmcl3d [class8@seismicmstm 100001]$ cp mymachines ../ [class8@seismicmstm 100001]$ ls ../ 100001 100002 100003 mymachines [class8@seismicmstm 100001]$ mpirun_rsh -hostfile ../mymachines np 96 pmcl3d We can also write this command into a script, then you can execute the script and all the jobs will be submitted togethe. The first step is open a new script using command emacs and edit the script , then check the script content with the command more. The details for the script please reference last example that I was showing before. [class8@seismicmstm EWF]$ emacs run_job.sh [class8@seismicmstm EWF]$ more run_job.sh
cd 100001 mpirun_rsh -hostfile ../mymachines np 96 pmcl3d cd ../100002 mpirun_rsh -hostfile ../mymachines np 96 pmcl3d cd ../100003 mpirun_rsh -hostfile ../mymachines np 96 pmcl3d

[class8@seismicmstm EWF]$ chmod +x run_job.sh [class8@seismicmstm EWF]$ run_job.sh Now, Id like to introduce a little more information about cluster and running a job. The cluster seismicmst has 32 nodes and each nodes has 4 processors, so, totally it has 128 processors. In our example, each job needs 96 nodes, so we can only run one jobs at each time. And other jobs you have submitted will be in the waiting line, these jobs will run one by one in this situation. If your job only needs 30 nodes, four jobs can run together at same time if there is

26

no other people use this cluster.But most time there will be lots of more users in the public cluster, so you have to be patient to wait. Most clusters prefer to run these jobs that require little time and less node first. But the waiting time on a cluster is also important. For these jobs that require longer time and more nodes, the job that has longer waiting time will run first. Acturely, cluster seismicmst is a small cluster. For most pubilic cluster, they have much more nodes than this one, surely, they also have much more users. At this situation, writing a script prefered by cluster become signaficient important,specially at busy time. Any user wants their jobs can be can as soon as they submitted them. But that is impossible. The capebility of each cluster is limited, when the totally require nodes is more than its design, some jobs need to wait at this situation. The tureth is the requirement is always exceed it design , so shortening the waiting time is become our first aim when we write a script. There are some method to achive this goal. First one is to reduce the require time. For this way you can separate your long job script into seveval shot script and each scrpt run less jobs. Probally you can get more chance to your job sooner than you write all the submitted jobs together. Maybe you will say since the require time smaller is better why dont we just write each job in a script? The reason is for some cluster they have limit for job script you have sent. For example cluster interpid, it has limited for 10 jobs or scripts in the waiting line. But for some cluster like ABES there is no limited. To write each job in a script is a good idea in this kind of cluster. Another choice for you to short the waiting time is to reduce the nodes number that you required, but be careful. The node number you required is based on the model you have built. This is a system project. Before you built you model you got tons of infector to think about like you shot style that affact how deep the wave can be transmitted, the space between shot and receivers sapce and the precision you required for you job and speed of your calculation you except and so on. In our example, we built our model into 240*60*180 grids, the grid space is 200 meters. We require 8*2*6 nodes and every node calculate 30 grids (these number you can find at /seismic04/class8/HALFSPACE1/VM1/XYZGRDfile). If you change your model, lets say 100 meter for grid space, then the totally grid number will become 480*120*360, If you like to calculate around 60 grids in each node you may need 8*2*6 nodes. Or you want change the grid numbers calculated in each node. It is variable for you to build your model. Less grid numbers calculated in a node do faster than more grid numbers calculated in a node. For my experience, keep the grid numbers calculated in each node in 30 to 60 is more efficient. For the fair any user should wait on the cluster . What we can do is to write a smart srcipt that require less time and less but enough nodes. Anyway, the rules in different cluster may a little bit different, if you want to write an efficacy job script you need to do a little survey for your own cluster.

27

3.3 RGT(receiver strain green tensor) Calculation

RGT simulation means we presume any one station as a source and look other sources and stations as receivers then calculate wave-form from specified source to other receivers and every time we calculation three strains from X,Y,Z direction. That means if we have three source and two receivers in EWF, we need to look as this receiver as a source and other source as stations. So we need to calculation 3*3=9 times to get RGT result. the Here we name this job directory as RGT that means receiver strain green tensor. We use the same calculation method and call the same python code pmcl3d with Forward simulation. the aim to do this RGT simulaiton is get kernel through convolution by two wave field. Another benefit for us to calculate the RGT simulation is to check the resolve of our simulation. Since we do the same calculation with the same code between the same two object, the result should be the same. We can prove that through comparing two wave-forms. If your calculation is correct the two wave-forms will superpose. If they are not you can say there is something wrong about your calculation either EWF calculation or RGT calculation or both. The main step to generate RGT directory and calculation are same with EWF simulation. But still there is a little bit difference. This step is the inverse of forward simulation. To make it clear, I give some of the job flow: [class8@seismicmstm JOBS]$ /seismic04/class8/F3DT/F3DT_FD_1.1/scripts/pmcl3d_setup.py param_HS.dat RGT 1 3 [class8@seismicmstm JOBS]$ cd ../RGT [class8@seismicmstm RGT]$ ls
IN.RC01.1 SF.RC03.1 IN.RC01.2 IN.RC01.3 SF.RC03.2 SF.RC03.3 SF.RC02.1 SF.RC02.2 SF.RC02.3

[class8@seismicmstm RGT1]$ cd IN.RC01.1 [class8@seismicmstm IN.RC01.1]$ ls

IN3D input_rst output_dyn output_sfc IN.RC01.1.src IN.RC01.1.src.inp output_vlm pmcl3d SETTING output_ckp

[class8@seismicmstm IN.RC01.1]$ diff IN3D ../../EWF/100001/IN3D

1c1 < 1 represent non-earthquack source --difference at line 1st --line 1st at IN3D is 1

28

--> -1 --line 1st at ../../AWF/100001/IN3D is -1 represent earthquack source 9c9 --difference at line 9th < 801 --line 9th at IN3D represent time step is 801 --> 150 --line 9th at ../../AWF/100001/IN3D represent time step is 150 44c44 --difference at line 44th < '/seismic04/class8/HALFSPACE1/RGT1/IN.RC01.1/IN.RC01.1.src' --> '/seismic04/class8/HALFSPACE1/EWF1/100001/100001.src' 52c52 --difference at line 52nd < 801 --line 52nd at IN3D represent time step is 801 --> 150 --line 52nd at IN3D represent time step is 150

[class8@seismicmstm IN.RC01.1]$ wc IN.RC01.1.src [class8@seismicmstm IN.RC01.1]$ 802 4809 77734 IN.RC01.1.src [class8@seismicmstm IN.RC01.1]$ head IN.RC01.1.src

30

30 121 0.9581078E-07 0.0000000E+00 0.0000000E+00 0.0000000E+00 0.1097857E-05 0.0000000E+00 0.0000000E+00 0.0000000E+00 0.6235339E-05 0.0000000E+00 0.0000000E+00 0.0000000E+00 0.2373961E-04 0.0000000E+00 0.0000000E+00 0.0000000E+00 0.6907810E-04 0.0000000E+00 0.0000000E+00 0.0000000E+00 0.1657250E-03 0.0000000E+00 0.0000000E+00 0.0000000E+00 0.3443269E-03 0.0000000E+00 0.0000000E+00 0.0000000E+00 0.6407288E-03 0.0000000E+00 0.0000000E+00 0.0000000E+00 0.1093363E-02 0.0000000E+00 0.0000000E+00 0.0000000E+00

0.0000000E+00 0.0000000E+00 0.0000000E+00 0.0000000E+00 0.0000000E+00 0.0000000E+00 0.0000000E+00 0.0000000E+00 0.0000000E+00

0.0000000E+00 0.0000000E+00 0.0000000E+00 0.0000000E+00 0.0000000E+00 0.0000000E+00 0.0000000E+00 0.0000000E+00 0.0000000E+00

[class8@seismicmstm IN.RC01.1]$ ]$ mpirun_rsh -hostfile ../../EWF/mymachines np 96 pmcl3d The job directories of forward and inverse simulation have the same file organization. The only different is the IN3D file and IN.RC01.1.src file as I showed in the last example. The reason is in the data progressing, there are lots of data we got through geophones, most of time we do not need that much, so we chose a filter to delete those data that we do need and some noise. And for

29

the source, no matter this is earth-quack source, explosive source or some other type source, people try to chose a source time function to simulation the source energy distribution. So here, for forward simulation we chose a source tine function src and for the inverse simulation we chose a 3-Hz filter as the green tensor filter. Here in our example this is a low-pass filter that means it will filter waveform that frequency more that 3 Hz. You can chose different filter depend on different situations. [class8@seismicmstm IN.RC01.1]$ ]$ cd ../../JOBS [class8@seismicmstm JOBS]$ ]$ more param_HS.dat
SOURCE TIME FUNCTION = /seismic04/class8/F3DT/F3DT_FD_1.1/templates/src GREEN TENSOR FILTER = /seismic04/class8/F3DT/F3DT_FD_1.1/templates/stf_grn_3Hz.dat ...

3.4 To generate synthetic seismograph

It is easy to generate synthetic seismograph since we have already calculated the EWF or RGT. The python code is stored in script directory like other python code. Similarly, we need some input files for this python code. The first one is param_HS.dat ; the second input is the source list ; the third and the fourth input are the starting and the ending source number that you want to rotate; the fifth input is the station list; the sixth and the seventh input is the starting and ending station number that you want to rotate. In the following example I just want to get the waveform between the first source and the first receiver. Steps as showing; [class8@seismicmstm JOBS]$ emacs ewf2syn_rotate.csh [class8@seismicmstm JOBS]$ more ewf2syn_rotate.csh
/seismic04/class8/F3DT/F3DT_FD_1.1/scripts/ewf2syn_rotate.py param_HS.dat eveinfo/evelist.foc 1 1 stainfo/stalist.foc 1 1

[class8@seismicmstm JOBS]$ sh ewf2syn_rotate.csh [class8@seismicmstm JOBS]$ cd ../SYN/ [class8@seismicmstm SYN]$ ls

SYN_EWF

[class8@seismicmstm SYN]$ cd SYN_EWF

30

[class8@seismicmstm SYN_EWF]$ ls
100001

[class8@seismicmstm SYN_EWF]$ cd 100001 [class8@seismicmstm 100001]$ ls

ewf2syn_100001.inp IN.RC01.box IN.RC01.rot rotsyn_IN.RC01.inp

[class8@seismicmstm 100001]$ wc IN.RC01.box

1054 3162 57970 IN.RC01.box

[class8@seismicmstm 100001]$ wc IN.RC01.rot

1054 5270 95914 IN.RC01.rot

[class8@seismicmstm 100001]$ more ewf2syn_100001.inp

/seismic04/class8/HALFSPACE/VM/XYZGRD /seismic04/class8/HALFSPACE/VM/HSGRDRECSTA /seismic04/class8/HALFSPACE/EWF/100001/output_vlm/SX96PVSTA /seismic04/class8/HALFSPACE/SYN/SYN_EWF/100001/SX96PVSTA 210 30 60 1 1054 0 30 30 60 [class8@seismicmstm 100001]$ more rotsyn_IN.RC01.inp 210 30 60 30 30 60 1054 /seismic04/class8/HALFSPACE1/SYN1/SYN_EWF/100001/IN.RC01.box /seismic04/class8/HALFSPACE1/SYN1/SYN_EWF/100001/IN.RC01.rot

We

can

see

that

through

the

output

file

in

directory

SYN_EWF/100001

file

ewf2syn_100001.inp and file rotsyn_IN.RC01.inp are the input files that the python code generate and call by itself. File ewf2syn_100001.inp is the input for python code to generate the waveform and file rotsyn_IN.RC01.inp is the input for python code to rotate the waveform to the coordinate. For the inverse waveform we use the same steps, showing as follow;

31

[class8@seismicmstm 100001]$ cd ../../../JOBS [class8@seismicmstm JOB]$ emacs rgt2syn_rotate.csh [class8@seismicmstm JOB]$ more rgt2syn_rotate.csh
/seismic04/class8/F3DT/F3DT_FD_1.1/scripts/rgt2syn_rotate.py param_HS.dat eveinfo/evelist.foc 1 1 stainfo/stalist.foc 1 1

[class8@seismicmstm JOB]$ cd ../SYN [class8@seismicmstm SYN]$ ls

SYN_EWF SYN_RGT

[class8@seismicmstm SYN]$ cd SYN_RGT [class8@seismicmstm SYN_RGT]$ ls

100001

[class8@seismicmstm SYN_RGT]$ cd 100001 [class8@seismicmstm 100001]$ ls

ewf2syn_100001.inp IN.RC01.box IN.RC01.rot rotsyn_IN.RC01.inp

3.5 Using Matlab software to check result

After we got the waveform we need to find a way to check its result, the easiest way is to draw it. So here we choose Matlab as a tool. As I mentioned in the last section, we got the forward waveform and inverse waveforms by different filter and source time function. For forward simulation, we use a source time function and the explored impulse together to get the forward waveform. For inverse simulation, we put a 3 Hz filter at the receiver to get the inverse waveform. If we want to compare these two waveforms we need to make sure we deal with two waveforms with same procedure. So in our example, we need to convolute the waveform that got from EWF with a 3 Hz filter and convolute the waveform that got from GRT with the source time function that we use at EWF waveform. Before we do the steps that I just mentioned, fist we need to download these IN.RC01.rot files to your local computer or somewhere installed Matlab software. Since these two waveforms have the same name, we need to change one of them to another name. in case one IN.RC01.rot file will replace another one. [class8@seismicmstm HALFSPACE1]$ cd SYN1/SYN_EWF/100001/ [class8@seismicmstm 100001]$ ls

32

ewf2syn_100001.inp

IN.RC01.box

IN.RC01.rot

rotsyn_IN.RC01.inp

[class8@seismicmstm 100001]$ mv IN.RC01.rot IN.RC01.rot_1 Now we have the two waveforms with different names. The follow step is download them to my local computer where I install the Matlab software. Steps are as showing, Xiaofeng-Zhangs-MacBook-Pro:~ xiaofeng$ mkdir EWF_GRT Xiaofeng-Zhangs-MacBook-Pro:~ xiaofeng$ cd EWF_GRT Xiaofeng-Zhangs-MacBook-Pro:EWF_GRT xiaofeng$ sftp class8@seismicmst.gg.uwyo.edu Connecting to seismicmst.gg.uwyo.edu... class8@seismicmst.gg.uwyo.edu's password: ******** sftp> get /seismic04/class8/HALFSPACE1/SYN1/SYN_EWF/100001/IN.RC01.rot_1 ./ sftp> get /seismic04/class8/HALFSPACE1/SYN1/SYN_RGT/100001/IN.RC01.rot ./ sftp> bye Xiaofeng-Zhangs-MacBook-Pro:EWF_GRT xiaofeng$ ls
IN.RC01.rot IN.RC01.rot_1

Before we write a Matlab code to compare these waveforms. I want to introduce a little more about sftp. SFTP represents SSH File Transfer Protocol that is a network protocol designed to provide secure file transfer and manipulation facilities over SSH. That means once you sftp to another cluster, your position is the aim cluster no longer the local machine or cluster. In our example I sftp to seismicmst.gg.uwyo.edu from my local computer and get file /seismic04/class8/HALFSPACE1/SYN1/SYN_EWF/100001/IN.RC01.rot_1 to directory EWF_RGT at my local computer. Command get is used to transfer files from aim cluster to our local place and command put is used to transfer files form our local place to aim cluster. Except for these two waveforms we still need to download file src and file stf_grn_3Hz.dat. Xiaofeng-Zhangs-MacBook-Pro:~ xiaofeng$ cd EWF_GRT Xiaofeng-Zhangs-MacBook-Pro:EWF_GRT xiaofeng$ sftp class8@seismicmst.gg.uwyo.edu Connecting to seismicmst.gg.uwyo.edu... class8@seismicmst.gg.uwyo.edu's password: ******** sftp> get /seismic04/class8/F3DT/F3DT_FD_1.1/templates/src ./ sftp> get /seismic04/class8/ F3DT/F3DT_FD_1.1/ stf_grn_3Hz.dat ./

33

 OK, lets write a Matlab code to convolute the waveforms separately and draw them.
% a matlab code for checking the reciprity and generate isolation filter(part 1 ) clear all close all rgt_file=load('IN.RC01.rot_1'); %read the RGT seismogram ewf_file=load('IN.RC01.rot'); %read the EWF seismogram src=load('src'); % read source time function filt=load('stf_grn_3Hz.dat'); % read filter reduce_rate=2; % setting parameter match with param_HS.dat sp_t=0.0142325*reduce_rate; % 0.00142325 is the time-interval got from file IN3D rgt_r=rgt_file(:,4); % read the fourth column of file IN.RC01.rot_1 ewf_r=ewf_file(:,4); % read the fourth column of file IN.RC01.rot x1=1:1:length(rgt_r); % length of file IN.RC01.rot_1 after decimate x2=1:1/reduce_rate:length(rgt_r); % length of file IN.RC01.rot_1 before decimate rgt_r2=interp1(x1,rgt_r,x2,'spline'); %interplation ewf_r2=interp1(x1,ewf_r,x2,'spline'); %interplation rgt_out=conv(rgt_r2,src); time function ewf_out=conv(ewf_r2,filt); filter %convolution rgt _2 with source % convolution ewf_r2 with 3 Hz

figure % draw a picture plot(rgt_out,'LineWidth',2) %draw rgt_out, the width of line is 2 hold on % wait for drawing another line in the same picture plot(ewf_out,'r'); % draw ewf_out, the color of the line is red rgt_outd=decimate(rgt_out,2); ewf_outd=decimate(ewf_out,2); % decimate back % decimate back

figure % draw another picture plot(rgt_outd,'LineWidth',2) % draw rgt_outd, the width of line is 2 hold on % wait for drawing another line in the same picture plot(ewf_outd,'r') % draw ewf_out, the color of the line is red

34

%======= below is used to generate isolation filter ======(part 2) isf_winP=load('isf_wins.txt'); % download file for i=1:length(isf_winP(:,1)) % loop start conv_out=fopen(['100001.IN.RC01.R.' int2str(i) '.isf'],'w'); % open a file and write isf=zeros(length(rgt_r),1) % setup a matrix lengt(rgt_r) row and 1 column isf(isf_winP(i,1):isf_winP(i,2))=rgt_outd(isf_winP(i,1):isf_winP(i,2)); figure(100+i) plot(rgt_outd,'LineWidth',2) hold on plot(isf,'r') fprintf(conv_out,'%f\n0.0\n0.0\n%f\n0.0\n',isf_winP(i,1)*sp_t,sp_t); fprintf(conv_out,'%d\n%E\n0.0\n0.0\n',length(rgt_r),isf_winP(i,2)*sp_t) ; for i2=1:length(isf) %second loop start fprintf(conv_out,'%f\n',isf(i2)); end %second loop end fclose(conv_out); end % loop end % %

For this matlab script, it has been separate to two parts. The first part is to do the convolution and draw the result. In this matlab script I download these files that I need by using command load. And then I interpolated the waveform because we decimate them when we calculated the waveforms that help us to save some storage room and calculate time. The most important step here is convolution. As we mentioned in the last paragraph, here we convolute waveform got from rgt with source time function and convolute waveform got from ewf with 3 Hz filter by using command conv. The last step is to decimate the waveform back to the pattern just as we download from the cluster. The result is showing as following. In the picture 3.1.1 the red wave represents the waveform got from rgt after convolute with source time function and after interpolated. And the blue waveform represents the waveform got from ewf after convoluted with 3 Hz filter and after interpolated. I the picture 3.1.2 the color represents the same meaning with picture 3.1.1. The only different is in the sampling rate in picture 3.1.2 is decimated by 2.

35

 picture 3.1.1 (before decimate )

picture 3.1.12 (after decimate)

In these two pictures, we can see that the red waveforms and the blue waveform overlap together well. That means we got the correct result waveforms from ewf and rgt. I the second part of the matlab script, We separate these waveforms to three sections. And wrote these three section to three different isolation filter. The first section is P waveform that is the P-wave propagation (showing as picture 3.2.1). Here, capital represents direct wave and low case letter represents reflect wave. The second section is Pp waveform (picture 3.2.2). The third section is Ps waveform(picture 3.2.3). The second part will generate the isolation filter. IN this file, the first line is the beginning of P waveform section that read from isf_wins.txt; the forth line is the time interval after decimate; the sixth line is thell times step; the seventh line is the end of P waveform section that read from isf_wins.txt. Here. File isf_wins.txt is generated by yourself. After you draw the waveform that got from the convolution, you can cut the waveform to several part and write down the beginning and ending points of the section that you interested to a file named isf_wins.txt . In out example, I intercept three sections as P, Pp and Ps and I

36

wrote these six points to file isf_wins.txt into three lines and two columns. Command the first part and execute the second part you will get the following picture and three files named 100001.IN.RC01.R.1.isf , 100001.IN.RC01.R.1.isf and 100001.IN.RC01.R.1.isf .

picture 3.2.1

picture 3.2.2

picture 3.2.3 After the comparison we found that the waveform calculation is correct. The next step is to calculate kernel with these results such as file 100001.IN.RC01.R.1.isf' generated by this matlab code, output files in EWF and RGT directories. We will talk about details in next chapter.

37

Chapter 4

Frchet Kernel

4.1 Kernel calculation on cluster

Frchet kernel is the data sensitive kernel, which qualified the observed and synthetic seismograms, with respect to structural parameters. We calculate this kernel basic on the seismograms that got from EWF and RGT. Since we have already got the correct waveform checked by the matlab code in the last chapter, in this chapter what we need to do execute this kernel python code and download this files to software ParaView to see the result showing as the following picture. All the kernel calculation are be done in directory /seismic04/xzhang6/HALFSPACE/KER1/ that directory we has already create in the very beginning. Similar with the organization that we calculate the GRT , here we need to calculate kernel in different phase as we mentioned before. So here we need to create three directories. As showing as below, directory name 100001.IN.RC01.R.1 represents phase P between shot one and receiver one; directory name 100001.IN.RC01.R.2 represents phase Pp between shot one and receiver one; , directory name 100001.IN.RC01.R.3 represents phase Ps between shot one and receiver one. [xzhang6@seismicmstm JOBS]$ cd ../KER [xzhang6@seismicmstm KER]$ mkdir 100001.IN.RC01.R.1 100001.IN.RC01.R.2 100001.IN.RC01.R.3 For the python code that we use to calculate the kernel has already been compiled in the very beginning. Since this python code is functional so we need to write an input file to tell this python code about what we want to get. Here we can ask this python code to calculate AP(alpha phase), AQ(alpha attenuation), BP(beta phase), BQ(beta attenuation ), LP(lambda phase), LQ(Lambda attenuation ), MP(Mu phase), MB(Mu attenuation), RP(Rho phase), RQ(Rho attenuation). Let see an example of an input file.

38

[xzhang6@seismicmstm KER1]$ cd 100001.IN.RC01.R.1 [xzhang6@seismicmstm 100001.IN.RC01.R.1]$ emacs ker.inp [xzhang6@seismicmstm 100001.IN.RC01.R.1]$ more ker.inp
/seismic04/class8/HALFSPACE1/VM1/XYZGRD /seismic04/class8/HALFSPACE1/VM1/HSGRDREC /seismic04/class8/HALFSPACE1/VM1/HSGRD /seismic04/class8/HALFSPACE1/EWF1/100001/output_vlm/SX96PV /seismic04/class8/HALFSPACE1/EWF1/100001/output_vlm/SX96PVVEL /seismic04/class8/HALFSPACE1/RGT1/IN.RC01.1/output_vlm/SX96PV /seismic04/class8/HALFSPACE1/RGT1/IN.RC01.2/output_vlm/SX96PV /seismic04/class8/HALFSPACE1/RGT1/IN.RC01.3/output_vlm/SX96PV /seismic04/class8/HALFSPACE1/RGT1/IN.RC01.1/output_vlm/SX96PVVEL /seismic04/class8/HALFSPACE1/RGT1/IN.RC01.2/output_vlm/SX96PVVEL /seismic04/class8/HALFSPACE1/RGT1/IN.RC01.3/output_vlm/SX96PVVEL /seismic04/class8/HALFSPACE1/KER1/100001.IN.RC01.R.1/100001.IN.RC01.R.1 .isf /seismic04/class8/F3DT/F3DT_FD_1.5/templates/src /seismic04/class8/HALFSPACE1/KER1/100001.IN.RC01.R.1/ker 210 30 60 %grid numbers of the model 30 30 30 %grid numbers of each chunk 150 0.0144 2 %source time function information 0 0 6 0.2 0.5 %the first 0 represent no filter in this calculation. 4 % the forth component we used to do the calculate 5 % five frequencies we want to calculate 0.2 0.5 1.0 3.0 3.0 %frequency value that we want to calculate 30 -1 8816 400 %time and space buffer

In this input file, the first thirteen line is the files that the python code need to call when it calculate the kernel. The fourteenth line is where the output file should be store. The steps to generate the kernel is showing as follow, [class8@seismicmstm 100001.IN.RC01.R.1]$ mpirun_rsh -hostfile ../../EWF/100001/mymachines np 96 /seismic04/class8/F3DT/F3DT_FD_1.5/bin/ker_filter_par5 ./ker.inp The output file looks like this as following, [class8@seismicmstm 100001.IN.RC01.R.1]$ ls
100001.IN.RC01.R.1.isf ker000000000.AP ker000001004.LQ ker001001003.AP ker002001001.LQ ker003001000.AP ker004000004.LQ ker005000003.AP ker006000001.LQ ker007000000.AP ker007001004.LQ . . ker000000000.LP ker003001000.LP ker007000000.LP ker000001004.RQ ker004000004.RQ ker007001004.RQ ker001001003.LP ker005000003.LP ker002001001.RQ ker006000001.RQ

39

 ker.inp

[class8@seismicmstm 100001.IN.RC01.R.1]$ls ker*.AP |wc

96 97 2311

[class8@seismicmstm 100001.IN.RC01.R.1]$ls ker*.AQ |wc

96 97 2311

This files that we see in the up paragraph are binary files separate by 96 chunks. Before we calculate the kernel we need to connect this files together with another python code /seismic04/class8/F3DT/F3DT_FD_1.5/bin/ker_collect_bin and also we need an input file showing as bellow, [xzhang6@seismicmstm KER1]$ cd 100001.IN.RC01.R.1 [xzhang6@seismicmstm 100001.IN.RC01.R.1]$ emacs ker_collectbin_AP.inp [xzhang6@seismicmstm 100001.IN.RC01.R.1]$ more ker_collectbin_AP.inp
/seismic04/class8/HALFSPACE1/VM1/XYZGRD /seismic04/class8/ HALFSPACE1/VM1/HSGRDREC /seismic04/class8/HALFSPACE1/KER1/100001.IN.RC01.R.1/ker AP 2 2 2 5

In this input file, the first two lines are input files location, and the third line is the output file location and forth line AP is what you want to calculate about the kernel. Here we want to calculate alpha phase. There fifth line is the grid decimate rate. The sixth line is about how many frequency kernels do you want to calculate. In our example we will calculate five frequency kernels. To finished this part steps is showing as below, [xzhang6@seismicmstm 100001.IN.RC01.R.1]$ ker_collect_bin ker_collectbin_AP.inp [xzhang6@seismicmstm 100001.IN.RC01.R.1]$ ls
100001.IN.RC01.R.1.isf ker000000000.AP ker000001004.LQ ker001001003.AP ker002001001.LQ ker003001000.AP ker004000004.LQ ker005000003.AP ker006000001.LQ ker007000000.AP ker007001004.LQ . ker.inp ker_AP_001 ker_AP_002 ker_AP_003 ker_AP_004 ker_AP_005

In the first command line, I did not give the full directory of ker_collect_bin that is because we have already link this executable to the bin directory in the compile file. As you see we got five output files from ker_AP_001 to ker_AP_005. Each file has already connect the required parameter those for calculate kernels together. And each files name represents different kernel.

40

For example, file ker_AP_001 represent alpha phase kernel at frequency 0.2 Hz that is the first value at line 21st we set up at file ker.inp . Since this file generate at directory 100001.IN.RC01.R.1 , so this file represent 0.2 Hz, alpha phase kernel between shot one and receiver one. Logically, file ker_AP_005 represent 3.0 Hz, alpha phase kernel between shot one and receiver one. Actually, these ker_AP_00* files are not the data format that software ParaView can read. We need to convert these files to the format that software ParaView can read. There is a python code in the bin directory can do this job. Steps as following, [xzhang6@seismicmstm 100001.IN.RC01.R.1]$ asc2bin ker_AP_001 ker_AP_001.bin [xzhang6@seismicmstm 100001.IN.RC01.R.1]$ ls
100001.IN.RC01.R.1.isf ker000000000.AP ker000001004.LQ ker001001003.AP ker002001001.LQ ker003001000.AP ker004000004.LQ ker005000003.AP ker006000001.LQ ker007000000.AP ker007001004.LQ . ker.inp ker_AP_001 ker_AP_002 ker_AP_003 ker_AP_004 ker_AP_005 ker_AP_001.bin

4.2 kernel draft on software

Now, this file ker_AP_001.bin is the format that software ParaView can read. The next step is to download this file and draw the imaging with software ParaView. Xiaofeng-Zhangs-MacBook-Pro:~ xiaofeng$ mkdir kernel Xiaofeng-Zhangs-MacBook-Pro:~ xiaofeng$ cd kernel Xiaofeng-Zhangs-MacBook-Pro:kernel xiaofeng$ sftp xzhang6@seismicmst.gg.uwyo.edu Connecting to seismicmst.gg.uwyo.edu... xzhang6@seismicmst.gg.uwyo.edu's password: ******** sftp> get /seismic04/xzhang6/HALFSPACE1/KER1/100001.IN.RC01.R.1/ker_AP_001.bin ./ sftp> bye Xiaofeng-Zhangs-MacBook-Pro:EWF_GRT xiaofeng$ ls
Ker_AP_001.bin

Since we have already got the file. The next step is setting up parameters in software ParaView. Parameters are showing as below. Here, I want to emphasize Data Extent part in picture 4.1.1. These values are relative with model grid number and decimate rate. The three lines in the Data Extent box represent X direction, Y direction and Z direction by sequence. In X direction, we

41

built 240 grids and the decimate rate is 2. So in the first line we got extent data 1 and 120. So on with the next two lines. In the picture 4.1.2, the first blank in Normal box represent display from X cross-section; the second and third blank represent Y and Z cross-section separately. Fill in 1 in specified blank represent showing image in this specified cross-section. In our example setting we set up in the middle blank to see Y cross-section.

42

picture 4.1.1 property setting picture

4.1.2 slice display setting

43

The result kernel of picture 4.2.1 is X component for P wave alpha phase kernel between shot one and receiver one in a uniform halfspace. The very left and right point is receiver and shot location. We can find their value at file stalist.foc and evelist.foc in directory /seismic04/class8/HALFSPACE1/JOBS1/. Since both the station and source location beneath the earth surface, that is why the upper side of the image is blank. We call this kernel is bananadoughnut kernel because what is the shape looks like. In all the kernels, the white color represent zeros, warm color represent negative amplitudes indicating that a velocity increase leads to an advance in arrive time and an increase in amplitude; and cool colors represent positive amplitude indicating a velocity increase leads to a delay in arrive time and reduction in amplitude. Picture 4.2.2 is same kernel as picture 4.2.1 at Z component.

picture 4.2.1 P phase alpha kernel-X component

picture 4.2.2 Z component

Pictures 4.2.3 is X component for Pp wave alpha phase kernel between shot one and receiver one in a uniform halfspace; We got this kernel by do calculation in directory

44

/seismic04/class8/HALFSPACE1/KER1/100001.IN.RC01.R.2/. Pictures 4.2.4 is X component for pS wave alpha phase kernel between shot one and receiver one in a uniform halfspace; We got this kernel by do calculation in directory /seismic04/class8/HALFSPACE1/KER1/100001.IN.RC01.R.3/. Kernel in picture 4.2.3 is for Pp wave, directory P wave is almost same strong as reflect S wave. So you can see both swing of the kernel. In picture 4.2.3, directory P wave is almost same stronger as reflect p wave. So you can see both swing of the kernel. Kernel in picture 4.2.4 is for Ps wave, directory P wave is much stronger as reflect s wave. So you can only see the right swing of the kernel.

picture 4.2.3 Pp phase alpha kernel

picture 4.2.4 Ps phase alpha kernel

Until now, we finished all the processer for calculate a complete kernel. What I introduced in this user menu is the basic step and thoughts to use this F3DT software package. Hopefully that gives new users basic idea about this software and how to use LINUX system. To see the further information and theory please reference the publication Frchet Kernels for Imaging Regional Earth Structure Based on Three-dimensional Reference Models and full 3D Waveform Tomography: A Comparison Between the Scattering-Integral and Adjoint-Wavefield Methods on website http://geology.uwyo.edu/?q=Dr.%20Po%20Chen

45