Repeatability, Reproducibility and Youden Plots

Donald J Wheeler, Craig Award Paper, “Problems With Gauge R&R Studies”, 46th Annual Quality Congress, May 1992, Nashv
Youden, William John, “Graphical Diagnosis of Interlaboratory Test Results”, Industrial Quality Control, May 1959, Vol. 15, No.
Donald J Wheeler, Richard W Lyday., Evaluating The Measurement Process, Second Edition, SPC Press, 1988
Donald S. Ermer and Robin Yang E-Hok, “Reliable data is an Important Commodity”, The Standard, ASQ Measurement Socie
Ott, Ellis R., “Analysis of Means – A Graphical Procedure”, Industrial Quality Control, August 1967, pp. 101-109
Donald J Wheeler, “An Honest Gauge R&R Study”, Manuscript 189, January 2009. http://www.spcpress.com/pdf/DJW189.pdf
Donald Wheeler, “How to Establish Manufacturing Specifications”, ASQ Statistics Division Special Publication, June 2003, http
Hopkins, Wil G., “A New View of Statistics”, http://www.sportsci.org/resource/stats/ , 1997
Prond, Paul, and Ermer, Donald S., “A Geometrical Analysis of Measurement System Variations”, ASQC Quality Congress Tra
Morris, Raymond A., and Watson, Edward, F., “A Comparison of the Techniques Used to Evaluate the Measurement Process”,
Futrell, David, “When Quality is a Matter of Taste, Use Reliability Indexes”, Quality Progress, Vol. 28, No. 5, May 1995, pp. 81-

Method Comparison
Bland, Martin, J., Altman, Douglas, G., “Statistical Methods For Assessing Agreement Between Two Methods Of Clinical Meas
Dietmar Stockl, Diego Rodrıguez Cabaleiro, Katleen Van Uytfanghe, Linda M. Thienpont “Interpreting Method Comparison St
Importance of Sample Size by Incorporating Confidence Limits and Predefined Error Limits in the Graphic”, Letter to the Editor

Kappa
Futrell, David, “When Quality is a Matter of Taste, Use Reliability Indexes”, Quality Progress, Vol. 28, No. 5, May 1995, pp. 81-

Mcnemar's Test
Agresti, Alan, “An Introduction to Categorical Data Analysis”, John Wiley & Sons, 1996

Sensitivity and Specificity

Stuart Spitalinic, MD, “Test Properties I: Sensitivity, Specificity, and Predictive Values”; Hospital Physician September 2004
Leefland, Moons, Reitsma, Zwinderman, “Bias in Sensitivity and Specificity Caused by Data-Driven Selection of Optimal Cutof
Klotins, K and Martin, S and Bonnett, B and Peregrine, A, “Canine heartworm testing in Canada: are we being effective?”
Alfred Saah, MD, MPH and Donald R. Hoover, PhD, MPH, “Sensitivity” and “Specificity” Reconsidered: The Meaning of The Te
ongress, May 1992, Nashville TN, pp. 179-185.
ol, May 1959, Vol. 15, No. 11
Press, 1988
ASQ Measurement Society Newsletter, Winter 1997, pp. 15-30.

ress.com/pdf/DJW189.pdf
ublication, June 2003, http://www.spcpress.com/pdf/DJW168.pdf

SQC Quality Congress Transactions – Boston, 1993

he Measurement Process”, Quality Engineering, 11(2), 1998, pp. 213-219
, No. 5, May 1995, pp. 81-86

Methods Of Clinical Measurement”, The Lancet, February 8, 1986

ng Method Comparison Studies by Use of the Bland–Altman Plot: Reflecting the
raphic”, Letter to the Editor of Clinical Chemistry, 50, No. 11, 2004

, No. 5, May 1995, pp. 81-86

sician September 2004

Selection of Optimal Cutoff Values”, Clinical Chemistry 54:4 (2008)
e we being effective?”
ed: The Meaning of The Terms in Analytical and Diagnostic Setting.; Annals of Internal Medicine.
 Measurement Discrimination Study

Opening Distance mm

Measurement Discrimination Plot

100

90

80

se
2
70

60 r  1
sT
2
50

1 r
40

Dr 
30

1 r
20

10

0
0 10 20 30 40 50 60 70 80 90 100
Opening Distance mm

se = #DIV/0! r= #DIV/0!
sp = #DIV/0! Dr = #DIV/0!
sT = #DIV/0! se is #DIV/0! of the total variation

#DIV/0!

Dr should be > 7.0 to proceed.

Measurement Range Chart

12
10
8
6
4
2
0
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30
 REPEATABILLITY & REPRODUCIBILITY
Initial Check

Measurement Variation: Stability

Average Range

Range Control Chart

12

10

0
Operator 1 Operator 2 Operator 3 Operator 4
Product Average

Operator Reproducibility: Bias

11.4
11.2
11
10.8
10.6
10.4
10.2
10
Operator 1 Operator 2 Operator 3 Operator 4

Page 4 of 21
 REPEATABILLITY & REPRODUCIBILITY
Overall System R&R

Repeatability & Reproducibility

All Operators
1
Measurement B

0.9

0.8

0.7

0.6

0.5

0.4

0.3

0.2

0.1

0
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
Measurement A

Operator 1 Operator 2
Operator 3 Operator 4
1:1 Line Specification

Repeatability se = #DIV/0!
Reproducibility so = #DIV/0!
R&R sR&R = #DIV/0! #DIV/0! of the total variation
Total Variation sT  #DIV/0!

Dre = #DIV/0!
Dr R&R= #DIV/0!

Page 5 of 21
 REPEATABILLITY & REPRODUCIBILITY
Individual Operator Repeatability

Operator 1 Operator 2
25 25

20 20
Trial B

Trial B
15 15

10 10

5 5

0 0
0 5 10 15 20 25 0 5 10 15 20 25
Trial A Trial A

se = #DIV/0! se = #DIV/0!
Dr = #DIV/0! Dr = #DIV/0!

Operator 3 Operator 4
25 25

20 20
Trial B

Trial B

15 15

10 10

5
5

0
0
0 5 10 15 20 25
0 5 10 15 20 25
Trial A Trial A

se = #VALUE! se = #VALUE!
Dr = Dr =

Page 6 of 21
 REPEATABILLITY & REPRODUCIBILITY
Operator to Operator Comparison Plots

Operator 1:2 Operator 1:3

10 10

Operator 3
Operator 2

8 8

6 6

4 4

2
2

0
0
0 2 4 6 8 10
0 2 4 6 8 10
Operator 1 Operator 1

Operator 1:4 Operator 2:3

1 1
Operator 4

Operator 3

0.9 0.9
0.8 0.8
0.7 0.7
0.6 0.6
0.5 0.5
0.4 0.4
0.3 0.3
0.2 0.2
0.1 0.1
0 0
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
Operator 1 Operator 2

Operator 2:4 Operator 3:4

Operator 4

1
Operator 4

0.9 1
0.8 0.9
0.8
0.7
0.7
0.6
0.6
0.5 0.5
0.4 0.4
0.3 0.3
0.2 0.2
0.1 0.1
0 0
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
Operator 2 Operator 3
Page 7 of 21
Operator 4
1

Operator 4
0.9 1
0.8 0.9
0.8
0.7
0.7
0.6
0.6
0.5 0.5
0.4 0.4
0.3 0.3
0.2 0.2
0.1 0.1
0 0
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
Operator 2 Operator 3

Page 8 of 21
 R & R: Multiple Operators
Data Sheet
Part Name/Number
Characteristic Name
Equipment & Units
Date of Study
Tolerance Range
sT, If Known Round to: decimals
Enter Data in Shaded Cells You need at least two operators to perform a reproducibility study
You may want to adjust the graphical output to plot only the number of operators you had in your study
Operator 1 Operator 2 Operator 3 Operator 4 Unit
Unit Trial A Trial B Trial A Trial B Trial A Trial B Trial A Trial B Average
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

Page 9 of 21
 30
Avg Avg_1: #DIV/0! Avg_2: #DIV/0! Avg_3: Avg_4: #DIV/0!
Rng Avg R_Op1: #DIV/0! Avg R_Op2: #DIV/0! Avg R_Op3: Avg R_Op4: 0
0

Page 10 of 21
 Measurement Discrimination Study

Do not enter data into these dark shaded cells

System 1 System 2
UCL LCL Mean Difference Center UCL LCL
Measurement Discrimination Pass/Fail Data Kappa Test

Measurement A
Pass Fail
Pass #DIV/0! #DIV/0! #DIV/0!
Measurement B
Fail #DIV/0! #DIV/0! #DIV/0!
#DIV/0! #DIV/0!

K= #DIV/0!
Measurement B

Fail

Pass

Pass Fail

Measurement A

pObserved  pChance
K
1  pChance

pObserved = p A Pass ∧ B Pass + p A Fail ∧ B Fail

pChance  ( p A Pass ´ pB  Pass )  ( p AFail ´ pB Fail )

 Amount
Number of Sample Kscore 1
Number of Judges
Number of Catagories

0
Instructions
Step 1. Fill out yellow spaces under Amount then click make table
Step 2. Fill in yellow space in table with data
Step 3. Click Calculate Results
Enter data in lightly shaded cells

Desired alpha: 0.05

One or Two-Tailed Test? Two-Tailed
Difference to detect

1st Measurement
Pass Fail
Measurement

Pass
2nd

Fail

Two-Tailed p 0.0000 Exact Binomial Test

The difference is statistically significant

Proportion Difference

Method Comparison: Categorical Data

1.000

0.900

0.800

0.700

0.600

0.500

0.400

0.300

0.200

0.100

0.000
1 2 3
 1st Measurement
Pass Fail
Measurement

Pass
2nd

Fail

Pa1
Pa2
σΔe

delta 0
ucl 0.000
lcl 0.000
detect 0 0 0

Confidence α zα/2
0.8 0.2 1.282
0.9 0.1 1.645
0.95 0.05 1.96
0.99 0.01 2.576
0.999 0.001 3.291
 Reference Method
Pos Neg Totals
Test Pos 0
Method Neg 0
Totals 0 0 0

95% Confidence Limits

Low CL High CL
Sensitivity #DIV/0! % 1.0 99.0
Specificity #DIV/0! % 1.0 99.0

Kappa Positive Negative

Statistic #DIV/0! Likelihood ratios #DIV/0! #DIV/0!

False-positive rate ### % Positive predictive #DIV/0! %

False-negative rate ### % value
Prevalence ### % Negative predictive #DIV/0! %
Diagnostic accuracy ### % value
 Truth / Standard

Desired Probability Target

Pos Neg Minimum Parameter Confidence
Totals Parameter PPR or NPR Meets Minimum Level Pass/Fail

Positive
Pos 0 #DIV/0! 0.9 10.00% 95% FAIL
Predictive Value
True Positives False Positives
Result
Negative
Neg 0 #DIV/0! 0.9 10.00% 95% FAIL
Predictive Value
False Negatives True Negatives

Totals 0 0 0

Parameter Sensitivity Specificity Prevalence

#DIV/0! #DIV/0! #DIV/0!

Desired Minimum
0.99 0.99
Sensitivity or Specificity

Probability Parameter
1.00% 1.00%
Meets Minimum

Target Confidence Level 95% 95%

Pass/Fail FAIL FAIL

 Gold Standard Test
R1 R2 R1 R2 Gold Standard
P N
P 0 0
Test
N 0 0
Sens Spec
#DIV/0! #DIV/0!

Percentages Sens Spec

Agreement #DIV/0! #DIV/0!
Within Gold Standard #DIV/0! #DIV/0!
Within Test #DIV/0! #DIV/0!
Between Tests #DIV/0! #DIV/0!

Enter P for Positive or N for negative

NOTE: Do not copy and paste results

100.00%
90.00%
80.00%
70.00% Between Tests
60.00% Within Test
50.00% Within Gold Standard
40.00%
Agreement
30.00%
20.00%
10.00%
0.00%
Sens Spec