Home What We Do Deeper Insight Contact

The truth about estimation #5 Nothing

is wrong with my model!

Posted by, Scott Dunham on October 3, 2017

So, you think you understand the geology and your data is all ready to go. Youve spent days, Search
weeks, months drilling, sampling, logging and interpreting. The CEO is breathing down your
bosss neck demanding an immediate update so it can be released to the market. After all those Recent Posts
shareholders are waiting to see exactly what their money has bought! The pressure is on. Its time
The truth about estimation #5
to start estimating. Ready to turn on the old computer and feed data and parameters in to
Nothing is wrong with my
generate an estimate?
model!
The truth about estimation #4
Data + Parameters = Estimate. What
How to interpret a drill
program (for mine and
Despite all the pressure and the demands for real visceral outcomes, it pays to slow down and do
resource geologists)
the hardest part of any resource geologists job Thinking.
The Reconciliation Myth #9

In many ways, we need to treat our modelling efforts like safety management. Every single on- When in doubt. scream and

the-job safety program Ive encountered over the years has had a similar thread. The first couple shout

of steps are to stop and think. Avoid rushing in and take that extra breath. Observe the The truth about estimation #3

environment, consider the task, look at the tools at hand and assess the risk. We focus on ensuring How to log a drill hole (for

people avoid injury (or worse). We focus on ensuring people go home in the same state of health as mine and resource geologists)

they had when starting work. And for good reason no one should have to face the prospect to The Reconciliation Myth #8

being injured just to earn a quid. Echoes in the wells of silence.

I like to apply the same concepts to my models. The better my thinking the better the model. I need Recent Comments
to stop, assess the situation and make appropriate decisions. Only then is it worth tackling the admin on The Reconciliation
estimate itself. Myth #4 The Day-2
Conundrum (with apologies
to my accounting friends)
mark gabbitus on The
Reconciliation Myth #4 The
Day-2 Conundrum (with
apologies to my accounting
friends)
admin on The Reconciliation
Myth #1 Factors
You see sometimes our thinking can be fatally flawed. While we like to think of ourselves as
Old Tom on Culture and
rational beings the truth is we are more rationalising than rational. We can convince ourselves
Success
just about anything is true if we put our minds to it. Let me give some examples
Crisis calls Old Tom on The
Reconciliation Myth #1
I recall a rocky start to the consultant-client relationship on one occasion. This particular client Factors
was in the process of recommissioning an open pit operation after a long and arduous feasibility
study. They were having some serious problems. Despite every effort, the expected grade (from Archives
the resource model) simply wasnt presenting at the ore treatment plant, or in grade control for
October 2017
that matter. I was engaged by the COO as part of a larger cross-discipline team to investigate the
September 2017
root cause of their grade shortfall and identify corrective actions.
August 2017
July 2017
After the investigation (which was fairly simple) I walked into a meeting with the CEO, the site
June 2017
General Manager, the Chief Geologist and the rest of the investigation team. The CEO kicked off
May 2017
the meeting by saying, Right we know theres nothing wrong with the resource model what is
going on?
Categories
Hed fallen into one of the biggest cognitive traps around. A form of group think. The company Capture

had spent millions on resource drilling, sampling, data validation, consultants. They had engaged Climate

industry experts to help with the modelling and estimation. They had gone well beyond what Commit

would be considered industry standard practice in terms of data collection and validation. How Consider

could the model possibly be wrong? He had a competent Chief Geologist and he had faith in the Create

rest of the geology team. They had done an excellent job it was all documented and logically Decide

presented. There couldnt possibly be a problem with the geology or resource estimation. Or so he Deliver

thought. Develop
Discover
This case of group think was one of the worst I have ever come across. It ruined several careers and Overview
very nearly ruined the organisation. And, it was all because of a combination of culture and belief. Uncategorized
The Chief Geologist, the Feasibility Study Manager and the Chief Project Development Officer
were all very opinionated and strong personalities. They formed a solid front facing off against Subscribe to Blog via
the rest of the organisation. They were the future. They were cleverer, more intelligent and more
Email
business savvy that everyone else. They deserved success. No, they simply were success. In the face Enter your email address to
of that sort of aggressive stance anyone who criticised their work was quickly shut down (and subscribe to this blog and
often removed!) This attitude even impacted the CEO. The message he received was We are the receive notifications of new
best you employed us to do a job and we are getting it done. We are the experts and you can posts by email.
(must?) rely on us.
Email Address
Now, Im all for self-belief and self-confidence. You cant be successful if you dont have faith in
your abilities. But and its a big but being overly confident can be more dangerous. Particularly
Subscribe
if it leads to dismissively ignoring the opinion of others, or selectively hearing only ideas and
opinions that reinforce your own.

This is exactly what happened for this client.

Remember this was a project that was re-starting an old operation, changing what was a selective
underground, high-grade mine into a bulk, low grade open pit. During the days of underground
mining the operation had a reputation for consistently over producing. The ore treatment plant
produced much more metal than was estimated in the resource model.

This history of under estimating and over producing ended up

You might think that
pervading the entire geological effort at the site. Every aspect
under estimating is a good
from drilling to estimation was aimed at proving the same effect
thing. Id argue its as bad
would occur in the new open pit. Everyone expected their
as over estimating. It leads
resource modelling and estimate would be biased low compared
to bad practice and
to the real production. The geology team went looking for
reliance on doing better.
evidence to support this hypothesis. And they found that
It can lead to sterilising
evidence.
what should be economic
mineralisation. It can lead
They found:
to bad operational and
 The closer the drill hole spacing the higher the grade of the financial design, over (or
mineralisation; under) capitalisation.
There was a bias between the reverse circulation (RC)
samples and the diamond drill hole samples. The diamond drill holes were always lower
grade than the RC;
In several test pits excavated during the feasibility study the grade of the bulk samples was
higher than the expected (modelled) grade.

It was a pretty conclusive story. The grade during mining and ore treatment was going to be
higher than expected.

Consequently, the grades in the resource model were factored up

Unfortunately, the
(increased) after estimation to reflect the evidence collected by
conclusions drawn from
the geology team. In some cases, the factored grade was 40%
these observations were
higher than the estimated grade.
entirely incorrect. In
incomplete understanding
And everyone from the CEO to the project geologists believed this
of sampling and statistics
was the right thing to do. After all the old underground mine
coupled with a self-
always produced more metal than predicted didnt it?
serving bias lead to fatal

As I said, group think at its worst, but there was more going on. flaws. There was an
Several other cognitive biases and cultural issues in the mix. alternative and more

There was emotional investment, issues of salience, noble cause correct explanation which
corruption, and fundamentally flawed logic. There was selection was totally ignored.
bias and self-serving bias. Contact me for more info.

Here are some symptoms that made this obvious to an impartial observer:

1. No one questioned the application of very large factors, despite factoring being inconsistent
with industry norms. Arguments for factoring were based on poor understanding of
statistics and experimental design;
2. The operation was considered special and unique. Different to every other mine in the
world and therefore it needed a unique approach to sampling, interpretation and
estimation;
3. There was clear evidence of consultant shopping. Many consultants had been involved
with the project at one stage but their involvement tended to be short term with a low level
of repeat business. To me that points to either the consultant not wanting to be involved
(reputation risk) or the client not agreeing with the consultants recommendations;
4. Consultant reports were being selectively quoted. Some very high profile consultants were
engaged and wrote quite critical reports. As many consultant do however, they also
highlighted areas of good practice. These positive comments were selectively quoted in the
feasibility study while the critical points were not included;
5. Evidence that didnt fit the preconceived idea of under-estimated grade was excluded or
written-off as poorly executed. If the data didnt fit the belief it was ignored;
6. The operational production target (in metal produced) was fixed and announced to the
market well before the feasibility study was complete. That locked in the answer and the
organisation simply couldnt back down from its messaging;
7. The risk assessment (yes there was one) failed to evaluate the consequences of factoring the
resource estimate. At no stage was the unfactored estimate used for an alternative
evaluation. Incidentally, if the unfactored estimate had been evaluated the operational
design selected in the feasibility study would not have passed the organisations investment
criteria.

Possibly the worst aspect of this entire story was the lack of any real desire to examine any
alternative causes for the observed data. No one tried to think outside of the box. Sadly, the group
think and self-serving bias prevented any attempts to really understand what the data was
showing. It was a case of success at all costs the preconceived operational design and production
target drove the study process. Anyone with a dissenting perspective was quickly shut down.

Despite the flawed thinking the organisation and the operation survived. The investment never
performed to expectation and there was an extreme level of management attention required to
placate shareholders and stakeholders alike. It was pure luck that the metal price exceeded
feasibility study predictions And yet a few simple steps could have prevented such a poor
outcome and arguably resulted in a much more profitable operation and certainly one that was
more enjoyable to work at! For any preventative measure to work however, there must first be
recognition of need.

Its that safety mantra Stop and Think. A powerful message and one that should be applied to
resource modelling and estimation too.

More Failures
OK, thats a true but extreme example. Lest you think this sort of thing couldnt be common or
couldnt happen to you, here are some other examples:

Excalibur Mining Corporation

2009 resource estimate for Tennant Creek projects 3.6Mt @ 10.16g/t Au (1.17Moz).
Including 1.1Mt @ 18.0g/t Au Indicated Resource
2010 resource estimate for Tennant Creek projects 2.1Mt @ 4.1 g/t Au (0.28Moz). All
Inferred Resource
Downgrade attributed to errors in database including mine workings used for depletion
BMA Gold Limited
Twin Hills Area 2 resource downgraded from 277Kt @ 21.6 g/t Au (193Koz) to 214Kt @
10.9 g/t Au (75Koz)
Downgrade attributed to significant reinterpretation of the geology
Triggered 30% plunge in share price, early mine closure and appointment of
Administrator
Deep Yellow Limited
37% increase in resources announced January 2004 (Mikado operations)
April 2004 Deep Yellow reports grade control indicating reserve model underestimating
contained gold by 10%
May 2004 Deep Yellow reports recovered gold down 18% on expectations due to ore
continuity issues. Mining suspended.

The list goes on and its not limited to juniors or

explorers. Some of the largest mining companies in the
world have had similar issues. The BHP and Billiton
merger in 2001 arose from a string of poor investments
by BHP in the last 1990s including Hartley Platinum
(below expected head grade due to ground conditions
and dilution), Beenup mineral sands (high clay content
and abrasiveness). Rio Tinto inherited problems at the

Interested in how thought process Mineracao Cabacal operation in the 1990s (>70%

affect your life this is worth a read. decrease in reserves).

That pressure to complete a model and make an estimate can be overwhelming. Stopping and
thinking is sometimes the hardest thing in the world. However, it is one of the most valuable ways
to avoid costly mistakes.
 The moment one has offered an original explanation for a phenomenon
which seems satisfactory, that moment affection for his intellectual child
springs into existence; and as the explanation grows into a definite theory,
his parental affections cluster about his offspring and it grows more dear to
him. While he persuades himself that he holds it still as tentative, it is none
the less lovingly tentative and not impartially and intemperately tentative
TC Chamberlain (1897).

Prevention is Better than Cure

Hopefully Ive convinced you of the value of stopping and thinking. Great, but exactly what do you
think about? Theres not much point in taking the time and energy to prepare for your modelling
and estimation unless you have some understanding of what you should be considering. This
thinking space is one reason the JORC Code requires five years of relevant experience in the style
of mineralisation or type of deposit and in the activity, being undertaken. Five years actively
working with the types of problems and challenges presented by the deposit you are modelling
and in modelling itself. That five years isnt just about surviving, its about learning, seeing,
experiencing. Getting to know the levers and drivers of resource quality and economics. I mean,
theres nothing quiet as educational as working in an operating environment, being responsible
for delivering outcomes and living through the inevitable differences between your predictions
and reality.

Each style of mineralisation has its own unique modelling challenges. Each estimation approach
has its own foibles. With that in mind however, there are common themes. Common aspects you
should stop and think about before you push the button on that old PC. Here are just a few you
never know they might just help you avoid group think and self-serving bias and prevent yet
another resource-related project failure.

Idea #1.
Remember its a model. As George Box famously said, All models are wrong, but some are
useful. The one thing you can be certain of is that your model will be wrong. Its a question of how
wrong is it, and in what way is it going to be wrong? Your first stop-think point is to understand
what your model is going to be used for and who is going to use it? I think intimate knowledge of
where your model fits is essential to good modelling. Take, for example, the difference between
Rutherford-Bohr atomic model and Erwin Schrdingersquantum mechanical atomic model.
Neither model is correct as we currently understand but they each have useful predictive
properties. Its the same with your resource model. You need to understand what you are trying to
predict and why. Understand the critical properties what matters and what is less important.
This includes global vs. local precision and accuracy, the importance of extrema and distributions
vs. averages and the spatial resolution required.

Idea #2.
Go back (again) to the geology and the geological data. Check if it is fit-for-purpose for the type of
model you need. What are the strengths and weaknesses inherent in the data? Double check have
you excluded any data because they dont fit your preconceived ideas? Have you excluded or
ignored observations that seem inconsistent with the other data? If so, stop-think. Make sure that
data really is worthy of exclusion avoid a selection bias. Just because you dont understand what
the data is telling you doesnt mean the data is wrong. Maybe you need to change your mental
model. Watch out for those everyone knows moments. After all, once upon a time everyone knew
there was no such thing as a black swan.

Idea #3.
Stop-think about what the geology is telling you. Ok, you are probably working with a genetic
model maybe intrusion-related gold or a porphyry copper system and you need to be a bit wary
of pigeon-holing your thinking based on what the genetic model claims, but If you are going to
work with a genetic model then use it to its fullest. What does the model predict in terms of metal
distribution? What controls grade, texture? What elements are associated and what are
contraindicated? What should the alteration and geochemistry look like? What about the
structure? Do you understand the fluid paths and traps? Do you understand the geometric
drivers? Are there useful fractal patterns, self-similar at different scales?

Idea #4.
Stop-think about that the statistics are telling you. Yes, the statistics. Even without going into full
modelling and estimation mode you should look at some basic stats and try and understand what
they mean in terms of your geological model, your domain model and your estimate itself. In the
example I discussed previously, not understanding what the statistics were signalling was one of
the fatal flaws that lead to those huge and unjustified factors. Hint, 90% of the time you will be
looking at skewed distributions, not a normal (Gaussian) distribution. That matters. Skewed
distributions are not as intuitive as normal distributions (as if any statistics were intuitive to most
of us!) Examine the comparative statistics of different sample types, different drill campaigns,
different geological units. Look for clustering and/or under-sampling. Look for areas that are
more or less variable. Think about support (and if you dont know what that means talk to
someone who does).

Idea #5.
Talk to your customers. Talk to the engineers, operators, managers, metallurgists. Find out whats
important to them and equally what they really dont care about. Ive lost track of the number of
models Ive seen where the first thing the planning engineer does is strip out 50% of the fields and
make up some of their own fields to suit their evaluation system. Ive even had one case where the
site engineer disbelieved the geology teams resource model so they did their own (independent)
estimate! If you understand whats important to your customer you are much better placed to be
able to provide a useful product. Thats at least half the battle won!

Idea #6.
Think about scale and resolution. Think about selective mining units. Think about the types of
decisions that will or will not be made on a block-by-block basis. Do you really need very small
(sub) blocks or will larger blocks/panels suffice? What is the likely difference (if any) between
block size and selection size. This can be a complex question, more so in underground operations.

Idea #7.
Ive said it before but it deserves its own stop-think action. Understand if your model should
represent averages or distributions (variability). Understand the impact of any smoothing that
goes with al weighted average estimation techniques (yes, that includes kriging). Do you need to
understand how variability may impact on downstream processes? In my opinion the answer to
that one is almost always yes. Do you need to understand the with-in-block grade/tonnage
distribution?

Idea #8.
Stop-think about the likely project economics before you start modelling the geology and grade.
What is the likely economic cut-off grade? How does that relate to any potential geology/grade
domains you may model? Is the cut-off close to the average of the deposit? Is the cut-off close to
any grade domain boundary you plan on modelling? These factors will impact on the quality and
usefulness of the model.

Idea #9.
Look at outlier values. Are they really outliers or do you simply not understand what the data is
telling you? Before you start adopting some top-cut, grade-cap or spatial grade restriction stop-
think. Are these apparent outliers simply due to data density? If you had more samples what
would these outliers look like? Go back to the geology and the logging are the outlier samples the
same style of mineralisation or something different? If they look different should they be in their
own separate domain even if you cant define it spatially?

And last Idea #10.

The most important idea of them all. Stop and think about what could be different. Assess the
degrees of freedom in your knowledge and data. Develop alternative views of the world and
consider the other 9 items in this list using those alternative views is there one that looks better
or as good as your original thinking? All too often we jump to a solution and fail to investigate
what could be. Consider:

What would the data tell you if every hole was 10% longer?
What would the data tell you if there was twice as much?
What would the data tell you if you eliminated 50% of the holes?
Have you adopted the parsimony principle the simplest explanation that fits all the data?
The explanation that requires the fewest deviations from the available observations, with
the least irrelevant detail?

I dare say if my client with the factoring problem had stopped and thought through these 10 items
they wouldnt have ended up in the mess they did. They managed to avoid being another tragic
company-killing resource estimation failure by the narrowest of margins. Its incumbent on all of
us as resource practitioners to learn from the lessons of the past and improve.

The definition of madness is doing the same thing over and over again and
expecting a different outcome.

Share this:

148

Like this:

Loading...

Related

The truth about estimation #2 - The Truth About Estimation #1 All samples are not equal the
Searching... Searching... Vested Interests question of legacy data
Searching...

Posted in Consider, Decide, Deliver, Develop Tagged Estimation, thinking

The truth about estimation #4 How to

interpret a drill program (for mine and
resource geologists)
LEAVE A REPLY

Your email address will not be published. Required fields are marked *

COMMENT

NAME * EMAIL * WEBSITE

Post Comment

Notify me of follow-up comments by email.

Notify me of new posts by email.

SD2 PTY LTD

my byline