Auditing Models - Verifying Through Review, Inspection, and Analysis

Auditing Models - Verifying Through Review,

Inspection, and Analysis

In this workshop we will review the auditing process for a resource model. The building
of a valid model from sample data requires sound geological modeling, geostatistical
methodology, assumptions, and resource calculations.
This technical review can be carried out through Process Review, Visual Inspection, and
Statistical Analysis. The goal of the audit is to examine the data and processes used by the
original resource geologist, as well as to create an audit report that in itself is repeatable
and will assist others in understanding the model.
We will cover these topics:
1. Why are we auditing?
2. Issues with Audits
3. Process Review
4. Visual Inspection
5. Statistical Analysis and Comparison
Mintec, Inc. 6. Review Reporting
Global
1. Why are we auditing?
Mining Before we start working on expanding or planning any mining operation which has a
Software resource model, we need to verify the validity of said model.
Solutions
Since There are industry standards which we need to uphold, as well as investors which we are
1970 accountable to.
a. SOX compliance -The Sarbanes-Oxley Act of 2002 is a United States federal law
passed in response to a number of major corporate and accounting scandals.
These scandals resulted in a decline of public trust in accounting and reporting
practices. The Act also covers issues such as auditor independence, corporate
governance, and enhanced financial disclosure.

Page 1
Auditing Models - Verifying Through Review, Inspection, and Analysis

b. National Instrument 43-101 (NI 43-101) is a mineral resource classification and is a

national instrument for the Standards of Disclosure for Mineral Projects with the
Canadian Securities Administrators. The NI is a strict guideline for how public
Canadian companies can disclose scientific and technical information about
mineral projects.
c. Lending institutions require that a project undergoes a stringent review and
validation process including the data used as a basis for any planning process.

2. Issues with Audits

In an ideal setting, a resource model has been built in a transparent and clear way. The
methods used are clearly explained and obvious. Unfortunately there can be issues which
challenge the auditing process.
a. Limited time or budget - The lack of either of these can leave certain areas uncovered
with gaps in the verification and validation of the modeling process and final
outcome. It is extremely important to scope the extent of the audit being performed
and the available information to perform these tasks. The amount of time available,
and therefore the budget for the work, must be clearly defined before any work is
done to ensure the results of the audit are reached within these parameters.
b. An inadequate audit trail of the modeling process will present the auditor with a
huge challenge. The audit process would have to assume details, and a lot of time
and resources will have to be spent testing these assumptions to approximate the
final model. Ideally a clear audit trail will minimize the guess work.
c. Lack of standards in the modeling process also leaves the auditor at a disadvantage.
Having nothing to compare to as far as standard practices and methodologies will
slow down the audit. The need for standards while building a model ensures that
when the auditor is reviewing the work, there will be a structure in place.

3. Process Review
One of the very first steps in reviewing a resource model is to read the accompanying
report. Understanding the entire process, the assumptions, and the results is essential in
determining the validity of the final outcome. Other details to consider are changes in the
original database due to errors or missing data, as well as how and what items are stored in
assay, composite, and model files. All available resources should be listed in the report, so it
should be great place to start.
a. Report Mintec, Inc.
• The report accompanying the resource models will tell us about the Global
parameters used in interpretation, modeling, and resource estimation. A Mining
careful reading of this report can explain the organization of the database (i.e., Software
separate domains for lithology, alteration, and definition of mineral zones) Solutions
• Geological modeling assumptions and methods (i.e., what sections were Since
used and what explanation is given to the geology being interpreted). 1970
Running reserves on the model to make sure the volume matches the solid
volume could be a way for checking, by numbers, that solids have coded the
model as expected.

Page 2
 Auditing Models - Verifying Through Review, Inspection, and Analysis

• Geostatistical methodology. Studies and results will tell us what parameters

were used in building the model. The understanding and verification of these
parameters is vital to the auditing process.
• Assumptions. It is important to review the assumptions made by the modeler.
Assumptions about geology, grades, data sets (if any were excluded or treated
differently) should be considered.
• Resource calculations: for example, how do the reserves for the updated model
compare to old ones?

b. Multi-runs
After reading the report, we can open the MineSight® project itself and have a look
around. Open existing MineSight 3-D (MS3D) drillhole views, model views, and
solids, then open MineSight Compass™ (MSCompass) and take a look at the PCF
and all existing project files (assay, composite, survey, gridded surface, 3D block
models, gridded seam models, and maybe others). A great way to see what work
was done to create this model is to look at existing multi-runs, which if done well,
can give us exact information on the modeling process (Figures 1a through 1c).

Figure 1a. Opening existing

multi-runs.

Mintec, Inc.
Global
Mining
Software
Solutions
Since
1970

Figure 1b. Existing multi-run.

Page 3
Auditing Models - Verifying Through Review, Inspection, and Analysis

Figure 1c. Variables in multi-run.

• A detailed look at all procedures in the multi-run will give us all the details
needed to understand the modeling process and can answer many questions
about the methodology used. If the modeler was diligent, there will be
accompanying notes that will give us a good insight into the way the model
was built. Above is an example of an ideal multi-run and variable setup.
• Reproducibility/Parameter confirmation. The ability to rebuild a model from
scratch using the same procedures as the modeler will quickly confirm that all
parameters used are correct and have not been changed.

c. History and Audit Files (xxxxhs.dat and msaudit.txt)

These two text files can provide us with useful details about the project being
audited. The History file keeps a chronological record of all programs run (Figure
2). The Audit file records every entry in the MS3D Messages window and every run
of an MSCompass procedure (Figure 3).

Mintec, Inc.
Global
Mining
Software
Solutions
Since
1970

Figure 2. History file

(xxxxhs.txt).

Page 4
 Auditing Models - Verifying Through Review, Inspection, and Analysis

Figure 3. msaudit.txt.

d. Archive data, run files, and report files in an organized manner.

Most MSCompass procedures allow naming and adding extensions to output data
files, run files, and report files (Figure 4). Organized naming can be a great way to
keep track of the steps taken in the modeling process. This methodology of storing
files is not only applicable during the modeling process, but also essential in the
auditing process.

Figure 4. Choose meaningful extensions and filenames.

e. Document your audit! The importance of keeping a clear audit trail of your work
while reviewing the model can not be stressed enough. The ability to reproduce
any part of the review will save you time and will allow any changes in the review
process to be incorporated easily. As mentioned above in multi-runs, a record of all
procedures is essential. Also, keeping clear notes for all steps and parameters used
Mintec, Inc. in the review process will ensure a transparent audit.
Global
Mining
Software
Solutions
Since
1970

Page 5
Auditing Models - Verifying Through Review, Inspection, and Analysis

4. Visual Inspection
a. Sectional Views
Being able to compare and visualize different elements of a resource project at a
glance is a great way to get a feel for the modeling process that was used. Most
resource projects will have an already-determined set of planes from which the
geology was interpreted. Being able to inspect the original assay data, composites,
geological solids/polygons, and model on these planes can be a valuable way to
gather information and get a better understanding of the deposit.

Mintec, Inc.
Global
Mining
Software
Solutions
Since
1970

Figure 5. 3D Section setup.

Page 6
 Auditing Models - Verifying Through Review, Inspection, and Analysis

Figure 6. 2D Section setup.

Mintec, Inc.
Global
Mining
Software
Solutions
Since
1970
Figure 7. 2D Plan setup.

b. Drillhole Locations/Blocks compared to Topographical Surface. A comparison,

both visual and through CAD functions in MS3D, between the drillhole collar
locations and the topographical data will pin point problem areas in the drillhole
database. Similarly, a comparison for the block model and topography should be

Page 7
Auditing Models - Verifying Through Review, Inspection, and Analysis

examined. If there are discrepancies between these (as shown in Figure 8) then
decisions must be made as to what can be done after measuring the effect these
could have on the final outcome.

Figure 8. Drillhole collars and blocks compared to topography.

c. Comparing Geologic interpretation to Drillhole data. Spearing the drillholes with

the geologic solids and comparing the results visually and statistically can verify
the correlation between the interpreted solids and the original database.
d. Drillhole erratic deviations can be spotted visually by displaying drillhole data by
cutoffs as well as through statistical analysis.
e. Make contour maps to study the tendency of mineralization, if there is any, and
compare this to the modeling parameters (Figure 9).

Mintec, Inc.
Global
Mining
Software
Solutions
Since
1970

Figure 9. Contour maps.

Page 8
 Auditing Models - Verifying Through Review, Inspection, and Analysis

f. Solid Modeling Integrity. Gaps and overlaps in the interpreted solids can also
be spotted through visual inspection. Any such discrepancies can point out
weaknesses in the interpretation and could determine the action which must be
taken in order to determine how to correct these flaws.
g. Make sectional views and make sure solids match blocks after coding.
h. Create repetitive plotting procedures for checking data if these procedures are not
part of the project already. The ability to create plots in a simple and reproducible
way increases the efficiency of the audit as you work through it. You may need to
check the results several times so you want to be easily able to recreate the plots
i. Check the validity of solids using the MS3D Surface | Verify function. Examine
solids to see if they intersect. Many times coding errors of blocks can be caused by
bad solids. Excessive intersection of solids can also result in coding errors.

5. Statistical Analysis and Comparison

Running standard statistical functions such as Histograms, Cumulative Probability Plots,
and Box Plots in MineSight Data Analyst (MSDA) will help us understand the population
with which we are dealing. We will be able to confirm if the assumptions about this
population on which the interpolation is based make sense. As mentioned in the Visual
Inspection part of the auditing process, these checks also give us a quick visual result that
can highlight anomalous data.
a. Customized reports. Use drillhole and composite statistics to confirm assumptions
in interpolation and to find possible errors shown below in Figure 10.

Mintec, Inc.
Global
Mining
Software Figure 10. Customized report.
Solutions
Since b. Box plots by rock give us a quick visual side-by-side representation of the
1970 distribution of our data by standard measurements (Mean, Median, 1st and 3rd
Quartiles, Observed Min and Max, among others) shown below in Figure 11.

Page 9
Auditing Models - Verifying Through Review, Inspection, and Analysis

Figure 11. Box plots by rock.

c. Cumulative probability plots allow us to compare populations graphically.

Mintec, Inc.
Global
Mining
Software
Solutions
Since
1970

Figure 12. Cumulative probability plot.

Making a triple plot of assay data, composites, and model blocks will show how the data
responds to the various operations. Theoretically the curves of the data should be identical.

Page 10
 Auditing Models - Verifying Through Review, Inspection, and Analysis

Example of multiple data plot.

d. Histograms can also give us a quick visual idea of the population with which we
Mintec, Inc. are dealing.
Global
Mining
Software
Solutions
Since
1970

Page 11
Auditing Models - Verifying Through Review, Inspection, and Analysis

Figure 13. Histograms.

e. Verify that variograms follow interpolation searches. Through MSDA we can run
variograms on the composite data and ensure that the parameters used in the
MSCompass interpolation procedures are valid. The nature of variography dictates
that we have a very clear idea of what parameters were run for variograms. A
slight difference in the setup (i.e., lags, directions, and nugget) could give us very
different results.

Mintec, Inc.
Global
Mining
Software
Solutions
Figure 14. MSDA variogram vs. MSCompass interpolation searches. Since
1970

Page 12
 Auditing Models - Verifying Through Review, Inspection, and Analysis

f. Use grade-tonnage curves to compare the results of different interpolations

schemes. In this example we compare Kriging (KRG), Inverse Distance Weighted
(IDW), and Polygonal Assignment (PLY) models.

Figure 15. Grade-Tonnage curves for different interpolation methods.

g. Point validation (M524V1) runs Ordinary Kriging or Inverse Distance Weighting

estimates for each composite using data from surrounding holes. Point validation is
not a test of block model validation but is used to verify the estimation parameters,
such as the variography. M525V1 (or MSDA) compares the original with the
estimated value using histograms and scatter plots.

Mintec, Inc.
Global
Mining
Software
Solutions
Since
1970

Page 13
Auditing Models - Verifying Through Review, Inspection, and Analysis

Figure 16. Point validation Using M524V1 and M525V1.

Mintec, Inc.
Global
Mining
Software
Solutions
Since
1970

Figure 17. Using MSDA to compare the results of

M524V1.

Page 14
 Auditing Models - Verifying Through Review, Inspection, and Analysis

h. Blasthole model vs. Exploration model Grade-Tonnage curves; correlation and

scatter plots. The ability to compare blasthole information against the interpolated
model can often point out weaknesses in the modeling results. If this data is
available, you can load it into a block model and run the same statistical tools
mentioned throughout the auditing process. The result of these analyses can tell
us how far off the model is compared to “real” data and can give us an idea how
strong the model is.

6. Review Reporting
All reserve reports from the original modeling process should be reproducible during the
review. Total ore and waste by cutoffs, by zones, and by classification should match the
original numbers and should be ‘reasonable’ when compared to your understanding of
the deposit.
a. Check in-situ reserves. Does it match a rough hand calculation or quick sectional
calculation in terms of volume and tonnage. Do different estimation methods (IDS,
Polygonal, Kriged) give comparable results for grade and tonnage.
b. Check mineable reserves. Do the in-pit reserves seem reasonable relative to the in-
situ reserves? Does the waste volume and stripping ratio look right?

Figure 18. Mineable reserves.

Check classification (proven, probable, possible) rules. The rules and logic used in
classification of ore are an important part of obtaining a valid model. These rules
should be reviewed and checked. Do the values for all classified material match the
totals for in-situ and/or mineable. Has the probable and possible been reduced to
Mintec, Inc. bulk up the proven reserves? Does the scheme used to do the classification seem
Global reasonable based on distance, number of composites, etc., for this type of deposit?
Mining
Conclusion
Software
All parts of the auditing process must verify that all the major elements normally associated
Solutions
with mine project work are fully addressed. Any discrepancies in either the visual process
Since or statistical reviews must be addressed, explained, and corrected before carrying on with
1970 any work on this model.

Page 15