Can You Measure It?

Planning a Successful Round Robin

Paul Schiffelbein
ASTM Committee Week
14 June 2006

Overview
Measurement system analysis (Round robin,
Interlaboratory Study (ILS), Interlaboratory
Comparison (ILC), Gage Study) description, definitions
and motivation
Round robin guidelines
Round robin logistics
Data Analysis
Precision and bias statement
2

Purpose of Measurement System Analysis

Ensure the measurement system has adequate

accuracy: precision and bias
Identify sources of measurement variation (and make
improvements, if necessary)
Compare several test devices, test methodologies, test
locations (linearity, bias, sensitivity)
Quantify measurement variability for reference

Measurement Process Variation

Observed Process Variation
Actual Process
Variation

Long-term
Process
Variation

Short-term
Process
Variation

bias
Simple

Measurement
Variation

Variation
within a
Sample

Bias
Linearity

Variation
due to
Gage

Short-term
Stability

Variation
due to
Labs,
Operators,
Devices,
Time

Long-term

precision

ISO 5725-1, ASTM E177

Accuracy: The closeness of agreement between a
test result and the accepted reference value
Accuracy, when applied to a set of test results, involves
a combination of random components and a common
systematic error or bias component

Bias: the difference between the expectation of

test results and an accepted reference value
Bias is the total systematic error as contrasted to
random error. There may be one or more systematic
components contributing to the bias.
5

ISO 5725-1, ASTM E177

Precision:
Precision The closeness of agreement between
independent test results obtained under
stipulated conditions.
Precision depends only on the distribution of random
errors and does not relate to the true value or the
specified value.

Precision can be decomposed into short- and longterm (or narrow and wide) components.
Repeatability and reproducibility are used to
quantify this concept.
6

ISO 5725-1
Repeatability: Precision under repeatability conditions

independent test results

the same test method
identical test items
the same laboratory
the same operator
the same equipment
within short time interval

Repeatability standard deviation: The standard deviation of

test results obtained under repeatability conditions
Repeatability limit (r): the absolute difference between two
test results obtained under repeatability conditions should be
less than or equal to this value

ISO 5725-1
Reproducibility: Precision under reproducibility conditions

independent test results

the same test method
identical test items
different laboratories
different operators
different equipment
longer time interval

Reproducibility standard deviation: The standard deviation of

test results obtained under reproducibility conditions
Reproducibility limit (R): the absolute difference between
two test results obtained under reproducibility conditions
should be less than or equal to this value
8

ASTM/ISO Usage
Observed Process Variation
Actual Process
Variation

Long-term
Process
Variation

Short-term
Process
Variation

bias
Simple

Bias
Linearity

Measurement
Variation

Variation
due to
Gage

Variation
within a
Sample

Short-term
repeatability
Stability

Variation
due to
Labs,
Operators,
Devices,
Time

Long-term

reproducibility
9

Gage R&R/Auto Industries Usage

Observed Process Variation
Actual Process
Variation

Long-term
Process
Variation

Short-term
Process
Variation

bias
Simple

Bias
Linearity

Measurement
Variation
Variation
due to
Labs,
Operators,
Devices,
Time

Variation
due to
Gage

Variation
within a
Sample

Short-term
repeatability
Stability

Long-term
reproducibility

Gage

10

Round Robin Guidelines

E 691 Standard Practice for Conducting an
Interlaboratory Study to Determine the Precision
of a Test Method
D 2904 Standard Practice for Interlaboratory
Testing of a Textile Test Method that Produces
Normally Distributed Data

11

Round Robin Guidelines

The design should be as simple as possible in order
to obtain estimates of within- and betweenlaboratory variability that are free of secondary
effects
Study should include a minimum of five
laboratories
A minimum of two operators should be used per
laboratory
When multiple instruments within a laboratory are
used, tests must be made on all equipment to
establish the presence or absence of the
equipment effect.

12

Logistics: D 885 Case Study

1 Motivation: The current version of D 885 was
written for traditional tensile testing machines.
Automated tensile testers are now being used in
high-tenacity fiber testing, and need to be included
in this standard. The current study will include
three types of automated testers, as well as
parallel testing on traditional tensile test devices.
The objective of this study is to quantify test
precision of traditional and automated testers, as
well as any bias seen between the device types.
The study will only address aramid materials.

13

Logistics: D 885 Case Study

2 Responsibility: Task group D13.19 (Tire Cord and
Fabrics) has overall responsibility of the ILS.
Dawn Caullwine (chair) will act as overall
coordinator for conducting the ILS.
The
coordinator will supervise the distribution of
materials and protocols to the laboratories and
receive the test result reports from the
laboratories.

14

Logistics: D 885 Case Study

3 Study Design: Nine materials will be tested on
each of three types of automated tensile test
devices. Two devices of each type will be used.
Yarn will be supplied in pre-twisted state for
testing. Untwisted yarn will also be provided to
Statimat labs, so testing can be performed both
on pre-twisted yarn, and yarn automatically
twisted by the test machine. Two laboratories will
also test the materials using traditional methods
for reference. Each of those laboratories will use
two operators.

15

Logistics: D 885 Case Study

Automated
Testers

Sigma 500

Lab A
Austria

Statimat

Lab B
Austria

Pre-twisted

Lab C
Europe

Statimattwisted

Uster

Lab D
USA

Pre-twisted

Lab E

Lab F

Statimattwisted

16

Logistics: D 885 Case Study

4 Materials: The ILS will include the following nine
materials (All samples are shipped twisted and
ready to test, except for Statimat laboratories,
which receive both twisted and untwisted samples):

Kevlar: 195 denier

Kevlar: 600 denier
Kevlar: 1140 denier
Nomex: 200 denier
Nomex: 1600 denier
Technora: 550 denier
Twaron : 840 denier
Twaron : 1500 denier
Twaron : 3000 denier

17

Logistics: D 885 Case Study

5 Test Determinations and Test Results: The
number of test determinations required for a test
result is specified in each individual test method.
For the purpose of this study, each laboratory will
make one hundred (100) determinations (breaks)
for each material.

18

Logistics: D 885 Case Study

5 Test Determinations and Test Results: The
following properties (and associated measurement
units) will be recorded:
N
Break strength (BS)
Elongation at break (EB) %
Modulus between 300 mN/tex and 400 mN/tex (MOD)
CN/tex
FASE @ 0.3% N
FASE @ 0.5% N
N
FASE @ 1.0%

Use nominal linear density for modulus calculation.

19

Logistics: D 885 Case Study

6 Details: The test method being studied is D885-03.
Specify the type of equipment used, including
manufacturer, model, and software program.
Samples should be conditioned as per D885 7.1 to
moisture equilibrium in an environmentally
controlled room for a minimum of 16 hours, at RH
55 +/- 2% and temperature 24 +/- 1 degrees C (72
+/- 2 degrees F).
Specific for Para-Aramid:
Gauge length: 500mm
Crosshead rate: 250mm/min. (50%)

Etc.

20

Logistics: D 885 Case Study

7 Data: Data should be entered into the Excel
workbook provided. Label the workbook with your
laboratory name, fill in the test data, and send the
completed file to:
Paul Schiffelbein
DuPont Engineering, Quality Management & Technology

paul.a.schiffelbein@usa.dupont.com
302/774-2417

21

Logistics: D 885 Case Study

22

Logistics: D 885 Case Study

23

A Better Way?: Internet Entry for F 2130

RR

Figure courtesy of Dr. Anugrah Shaw, UMES

24

Data Structure Guaranteed!

25

Logistics: D 885 Case Study

8 Data Analysis:
Material = Kevlar 1420d
Variability Chart for BF(N)
350
340
330
320
300
290
280
270

Instron

T T QRS

T FA

S p ru a n c e

T FA

S p ru a n c e

Ma in s ite

L e n zin g

T T QRS

T FA

250

S p ru a n c e B

260

S p ru a n c e A

B F(N )

310

Sigma500 Statimat Statimat PTUster

Laboratory within Device

26

Logistics: D 885 Case Study

8 Data Analysis:
Material = Kevlar 1420d
Variability Chart for BF(N)

15
10

Ins tron

TT QRS

TFA

Spruance

TFA

Spruance

Mainsite

Lenzing

TT QRS

TFA

Spruance B

5
Spruance A

Std Dev

20

Sigm a500 Statim at Statim at PT

U ster

Laboratory within Device

27

Repeatability and Reproducibility

Use between- and within-laboratory variance
components:

R = 2L + 2r
Where:

R is the reproducibility standard deviation

r is the repeatability standard deviation
L is the square root of the inter-laboratory
(device, operator, etc.) variance component
28

Logistics: D 885 Case Study

8 Data Analysis:
REML Variance Component Estimates
Random Effect

Var Ratio

Device&Random
Laboratory[Device]&Random
Residual
Total

0.6823116
0.4806723

Var
Component
66.35
46.74
97.25
210.36

Repeatability variance

Std Error

95% Lower

95% Upper

Pct of Total

62.607
26.848

18.903
19.480

1854.81
224.123

31.5
22.2
46.2
100.0

Reproducibility variance
(one estimate)

Reproducibility variance
(another estimate)
29

Logistics: D 885 Case Study

9 Precision and Bias Statement:
13. Precision and Bias
13.1 Interlaboratory Test Data An interlaboratory test was conducted in 2006 using
commercially available tensile testers from . Nine materials were included in
the study, and are listed in table _. The study structure is shown in Figure _.
Variance components were computed for individual tensile determinations, and are
summarized in Table _.

30

Logistics: D 885 Case Study

9 Precision and Bias Statement:
13.2

Precision - Repeatability and reproducibility deal with the variability of test

results obtained under specified laboratory conditions. Repeatability concerns the
variability between independent test results obtained within a single laboratory in
the shortest practical period. Those results are obtained by a single operator with
a specific set of test apparatus using test specimens (or test units) taken at random
from a single quantity of homogeneous material obtained or prepared for the
interlaboratory study (ILS). Reproducibility deals with the variability between
single test results obtained in different laboratories, each of which has applied the
test method to test specimens (or test units) taken at random from a single
quantity of homogeneous material obtained or prepared for the ILS.

31

Logistics: D 885 Case Study

9 Precision and Bias Statement:
Method repeatability is defined as the "maximum difference" that can
"reasonably" be expected between two test results obtained on the same material
when the test results are obtained in the same laboratory. Repeatability standard
deviation is taken to be the square root of the determination variance
component, and represents within-operator precision. Method reproducibility is
defined as the "maximum difference" that can "reasonably" be expected between
two test results obtained on the same material when the test results are obtained
from different laboratories. The total, or reproducibility, standard deviation, is
formed by taking the square root of the sum of intra- and inter-laboratory variance
components.

32

Logistics: D 885 Case Study

9 Precision and Bias Statement:
Table _ contains repeatability and reproducibility standard devations and
maximum critical differences for single determinations and specified averages of
determinations for the single operator case (repeatability), within-laboratory case,
and between-laboratory case (reproducibility).

Two values or averages of

observed values are considered significantly different at the 95% probability level
if the difference between them exceeds the appropriate critical difference in the
table.

33

Logistics: D 885 Case Study

9 Precision and Bias Statement:

13.3 Bias The procedure in this test method produces a test value that can be defined
only in terms of a test method. There is no independent referee method by which bias
may be determined. The test method has no known bias.

34

Summary
Explicitly state objectives for your study, and
make sure all participants understand them
Write protocol covering responsibilities, timing,
samples and sample handling, test equipment and
set-up, sampling plan, properties, units, etc.

Details are important! Be specific, explicit, and

discuss ahead of time with task group to avoid
surprises. Communicate!

Set up data collection to be as goof proof as

possible, and to facilitate subsequent analysis

35

Thank you

36