Applied Problem Solving and

Research Using Statistical

Methods with NIST Examples

Adam L. Pintar

September, 2016
Introduction
 About Me

Grew up in Kansas
Education
Pittsburg State University: Mathematics
: Statistics
Family

3/71
 About NIST

NIST’s mission is to promote U.S. innovation and industrial

competitiveness by advancing measurement science,
standards, and technology in ways that enhance economic
security and improve our quality of life.
NIST is the national metrology institute (NMI) for the United
States. As an NMI, NIST
Maintains primary measurement standards for the seven base
units in the SI system of units and for derived units
Offers calibration services and measurement standards to
support international trade
Develops new measurement technologies
The Institute for National Measurement Standards of the
National Research Council is the Canadian counterpart

4/71
NIST Campus

5/71
About SED

Churchill Eisenhart was the first

Chief
Originally Statistical Engineering
Laboratory
1947
Lola Deming was a founding
member too

6/71
More History

W. J. Youden (Right)
J. Cameron (Left)
Graeco-Latin Square
Enamel on metal

7/71
 Outline

Experiment Design
X-ray CT for detecting additive manufacturing (3D printing)
defects
Interactive discussion (choosing factors and levels)
Accelerated degradation of polymeric materials
Exploratory Data Analysis
Accelerated degradation of polymeric materials
Volumetric versus RECIST length tumor measurements
Standard reference material heterogeneity
Probability/Stochastic Modeling
Standard reference material heterogeneity
Distribution of peak pressure
Standard reference material values
Interactive discussion (choosing prior distributions)
Multidisciplinary projects
Localizing leaks of geologically sequestered CO2

8/71
Experiment Design
X-ray CT for detecting additive manufacturing (3D
printing) defects
 Goal

Best practices for measuring void size

11/71
 Question 1

What factors influence the measurements?

12/71
 List of Factors – Version 1

CT acquisition Reconstruction
1. Voltage 1. Algorithm
2. Current 2. Center of rotation
3. Filter type 3. Beam hardening correction
4. Filter thickness 4. Scattering correction
5. Magnification Artifact
6. Frame rate 1. Material
7. Number of images per projection 2. Flaw size
8. Detector type
3. Flaw shape
9. Pixel size
10. Scintillator type Image processing
11. Scintillator thickness 1. Smoothing filter
2. Thresholding algorithm

20 factors!

13/71
 Available Resources

Able to produce about 20 images

Need 220 ≈ 1m runs for a 2-level full factorial experiment
The closest we can get to 20 runs is a 220−15 fractional
factorial
32 runs
Main effects confounded with 2 factor interactions
No estimate of pure error

14/71
 List of Factors – Version 2

CT acquisition
1. Voltage (numeric)
2. Current (numeric)
3. Magnification (numeric)
4. Frame rate (numeric)
5. Number of images per projection (numeric)
Reconstruction
1. Algorithm (categorical)

15/71
 Design

## volt curr mag fr n_img alg

## 1 0 0 0 0 0 -1
## 2 0 0 0 0 0 1
## 3 -1 -1 1 1 1 1
## 4 1 1 -1 1 -1 1
## 5 1 -1 1 -1 -1 1
26−2 fractional factorial with 4 ## 6 -1 -1 1 -1 1 -1
“center” runs ## 7 1 1 1 1 1 1
## 8 -1 1 -1 -1 1 1
Main effects not confounded ## 9 1 -1 1 1 -1 -1
with 2 factor interactions ## 10 -1 -1 -1 -1 -1 -1
2 df pure error estimate ## 11 0 0 0 0 0 -1
## 12 1 1 1 -1 1 -1
Some information about 2 ## 13 1 -1 -1 -1 1 1
factor interactions ## 14 -1 1 1 1 -1 -1
## 15 -1 1 -1 1 1 -1
## 16 1 1 -1 -1 -1 -1
## 17 1 -1 -1 1 1 -1
## 18 -1 1 1 -1 -1 1
## 19 -1 -1 -1 1 -1 1
## 20 0 0 0 0 0 1

16/71
 Computation

R is a language and environment for statistical computing and

graphics. . . .

17/71
 2-level Fractional Factorial Designs in R

FrF2 function from the R package FrF2

factor_names <- c('volt', 'curr',

'mag', 'fr',
'n_img', 'alg')
my_design <- FrF2(factor.names = factor_names,
resolution = 4, ncenter = 4)

18/71
## volt curr mag fr n_img alg
## 1 0 0 0 0 0 0
## 2 0 0 0 0 0 0
## 3 -1 -1 1 -1 1 -1
## 4 1 1 -1 1 -1 1
## 5 -1 -1 -1 1 -1 1
## 6 1 -1 -1 1 1 -1
## 7 1 -1 1 -1 -1 1
## 8 1 1 1 -1 1 -1
## 9 -1 -1 -1 -1 -1 -1
## 10 1 -1 1 1 -1 -1
## 11 0 0 0 0 0 0
## 12 -1 1 -1 -1 1 1
## 13 -1 1 1 -1 -1 1
## 14 1 1 1 1 1 1
## 15 1 -1 -1 -1 1 1
## 16 -1 1 -1 1 1 -1
## 17 -1 1 1 1 -1 -1
## 18 1 1 -1 -1 -1 -1
## 19 -1 -1 1 1 1 1
## 20 0 0 0 0 0 0
## class=design, type= FrF2.center

19/71
design.info(my_design)$alias

## $legend
## [1] "A=volt" "B=curr" "C=mag" "D=fr" "E=n_img" "F=alg"
##
## $main
## character(0)
##
## $fi2
## [1] "AB=CE=DF" "AC=BE" "AD=BF" "AE=BC" "AF=BD" "CD=EF"
## [7] "CF=DE"

20/71
Interactive Discussion
Choosing Factor Levels
Accelerated Degradation of Polymeric Materials
Goal

Lab measurements

Field measurements

23/71
 Two Experiments

Laboratory
Precisely control/measure a few factors
Field
Environmental variation
Mimic building movement
Focus on Laboratory

24/71
 Lab Factors

Light intensity
Temperature
Humidity
Mechanical strain

25/71
 Design

Light intensity – Fixed at max

Temperature – 4 levels
Humidity – Fixed
Mechanical strain – 4 levels
42 full factorial

26/71
 50
45
40
Temperature (C)

35
30
25
20

0 5 10 15 20

Mechanical Strain (%)

27/71
Exploratory Data Analysis
Accelerated Degradation of Polymeric Materials
 Raw Data
0 20 40 60 80 0 20 40 60 80

1.2
Chamber 1
0.8

Chamber 2 Temp: 21 Temp: 21 Temp: 21 Temp: 21

Chamber 3 Strain: 0 Strain: 5 Strain: 11 Strain: 21
0.4

Chamber 4
0.0

1.2
0.8
Temp: 31 Temp: 31 Temp: 31 Temp: 31
Modulus Ratio

Strain: 0 Strain: 5 Strain: 11 Strain: 21

0.4
0.0
1.2
0.8

Temp: 41 Temp: 41 Temp: 41 Temp: 41

Strain: 0 Strain: 5 Strain: 11 Strain: 21
0.4
0.0

1.2
0.8
Temp: 51 Temp: 51 Temp: 51 Temp: 51
Strain: 0 Strain: 5 Strain: 11 Strain: 21

0.4
0.0
0 20 40 60 80 0 20 40 60 80

Exposure Days

30/71
Volumetric versus RECIST Length Tumor
Measurements
Experiment

32/71
 Raw Data
6

6
5

5
4

4
Mass (g)
3

3
2

2
2 3 4 5 6 20 25 30 35
Volume (cm3) RECIST (mm)

Color differentiates diapers

Expect straight lines with positive slopes

33/71
Standard Reference Material Heterogeneity
SRMs

35/71
SRM Certificate

36/71
Coal Bottle-to-Bottle Differences (Heterogeneity)
Bromine

4.4
Observations
Grand mean

4.2
4.0
3.8
mg/kg
3.6
3.4
3.2

2 4 6 8 10
Bottle

37/71
Probabilistic/Stochastic
Modelling
Standard Reference Material Heterogeneity
Bottle-to-Bottle Differences (Heterogeneity)
Bromine
Observations

4.5
Grand mean
Grand mean CI
Bottle mean CI
ANOVA p−value: 0.074
4.0
mg/kg
3.5
3.0

2 4 6 8 10
Bottle

40/71
 p-value

H0 : Bottles all the same

HA : At least one bottle different

anova(lm(raw_data$mg_kg ~ factor(raw_data$bottle)))

## Analysis of Variance Table

##
## Response: raw_data$mg_kg
## Df Sum Sq Mean Sq F value Pr(>F)
## factor(raw_data$bottle) 9 1.62004 0.180005 2.6253 0.07438 .
## Residuals 10 0.68565 0.068565
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

41/71
 Power
Probability of concluding at least one bottle is different when
that is the true state of nature

power.anova.test(groups = 10, n = 2,
between.var = c(0.5, 1, 1, 2), within.var = 1,
sig.level = c(0.01, 0.05, 0.15, 0.1) )

##
## Balanced one-way analysis of variance power calculation
##
## groups = 10
## n = 2
## between.var = 0.5, 1.0, 1.0, 2.0
## within.var = 1
## sig.level = 0.01, 0.05, 0.15, 0.10
## power = 0.08086632, 0.51463217, 0.77715565, 0.93728416
##
## NOTE: n is number in each group

42/71
Peak Pressure
Structural Design

44/71
 Experimental Data

0.0−1.0
Pressure
−2.0 −3.0

0 20 40 60 80 100
Time (s)

45/71
 Data Processing

3.0
2.5
1.5 2.0
Pressure
1.0
0.5
0.0

20 40 60 80 100
Time (s)

46/71
 Simulated Data

3.0
Pressure
1.0 2.0
Original Data Set
0.0

0 50 100 150 200

3.0
Pressure
1.0 2.0

Fake Data Set #1

0.0

0 50 100 150 200

0.0 1.0 2.0 3.0
Pressure

Fake Data Set #2

0 50 100 150 200

Time (s)

47/71
 Distribution of the Peak

Distribution of the Peak Value

1.5
Mean

1.0
Density

0.5
0.0

3.0 3.5 4.0 4.5

Peak Value

48/71
Uncertainty in the Distribution of the Peak

Distribution of the Peak Value

1.5
Mean

1.0
Density

Bootstrap Replicates
80% CI for the Mean
0.5
0.0

3.0 3.5 4.0 4.5

Peak Value

49/71
 R Package – Reference

https://github.com/usnistgov/potMax

50/71
Standard Reference Material Values
 Bayesian methods – high level

Two sources of information

Data
Subject matter expertise (Prior)
Bayes rule tells us how to combine them

52/71
 Hard Rock Mine Waste

Governor Basin, Colorado

53/71
 Ag (Silver) Raw Data

80
mg/kg
75
70

ICPMS Lab 1 ICPOES ICPMS Lab 2 INAA

54/71
 Posterior With Flat Prior
Ag (Silver)
0.08
0.04 0.06
Density
0.02
0.00

40 60 80 100 120
mg/kg

95% interval [55, 104]

55/71
 The Problem
80
mg/kg
75
70

ICPMS Lab 1 ICPOES ICPMS Lab 2 INAA

56/71
 Prior Information

Infomative
Flat
5
4
Density
3
2
1
0

0.0 0.2 0.4 0.6 0.8 1.0

Logarithmic Units

Quality control data (Log scale)

57/71
 Posterior With Informative Prior
Ag (Sliver)
0.00 0.02 0.04 0.06 0.08 0.10
Density

60 70 80 90 100 110
mg/kg

95% interval [66, 83]

58/71
 Interactive Discussion
Choosing Prior Distributions
Multidisciplinary Projects
 Goal

Detect and locate leaks of CO2 stored in geological formations

61/71
 Test Site in Montana

The data that were analyzed come from a similar site in Ft.
Wayne Indiana using the same equipment
62/71
 Likelihood at Ft. Wayne

Grid is clearly visible

Expected no signal, but found a strong one
63/71
 Collaborators

Zachary H. Levine (NIST)

Jeremy T. Dobler (Harris Corp.)
Nathan Blume (Harris Corp.)
Michael Braun (Harris Corp.)
T. Scott Zaccheo (Atmospheric and Environmental Research)
Timothy G. Pernini (Atmospheric and Environmental
Research)

64/71
 Reference

Levine, Z. H., Pintar, A. L., Dobler, J. T., Blume, N., Braun, M.,
Zaccheo, T. S., Pernini, T. G., ”The Detection of Carbon Dioxide
Leaks Using Quasi-tomographic Laser Absorption Spectroscopy
Measurements in Variable Wind,” *Atmospheric Measurement
Techniques*, **9**, 1627–1636, 2016, http://www.
atmos-meas-tech.net/9/1627/2016/amt-9-1627-2016.pdf.

65/71
 Summary

Three main toolboxes

Experiment design
Exploratory data analysis
Stochastic/Probabilistic modelling
Toolboxes interact with each other
Multidisciplinary teams

66/71
Questions

67/71
 Question One

Thank you Michelle T.

Synopsis
Purchase four physical standards for calibrating liquid
chromatography instruments
Combine the standards into a single new “secondary” standard
How to verify that the combination procedure does not change
the concentrations listed on the certificates

68/71
 Simple Graphical Approach
Quality control checks when measuring candidate SRMs

16
14
12
µg/kg
10
8

Certificate Measurements

69/71
 Formalization of the Simple Approach

H0 : No difference
HA : Some difference
Review paper
Rukhin, A. L., “Assessing Compatibility of Two Laboratories:
Formulations as a Statistical Hypothesis Testing Problem,”
Metrologia, 50, 49–59 (2013).
Potential problem
Can find strong evidence of a difference but not strong
evidence of equivalence

70/71
 Second Formalization
H0 : Difference is practically important
HA : Difference is not practically important
Reference
Anderson-Cook, C. M. and Borror, C. M., “The Difference
Between “equivalent” and “not different,”” Quality
Engineering, 28, 249–262 (2016).
Some modification necessary
16
14
12
µg/kg
10
8

Certificate Measurements

71/71