The Need for Experiments

Begin Module 1

Example of Experiment with One Factor

Module 5

Design and Analysis of Experiments

Module 4

Testing the Change

Module 3

Sequential Experiment s and PDSA

Module 2

The Need for Experiments

Module 1

Agenda

Rajiv Gupta
BITS Pilani

Design of Experiments

Control Charts: X-bar and R Charts

In control charts we plotted data points over time to

monitor whether a process was in control/stable
Only when an abnormality was observed in the
process output that we intervened to investigate the
cause of the abnormality and to correct it
This method is a passive method because we are not
taking proactive action to bring about a change in
the system. Rather we are reacting to specific
occurrences that are observed.
Only special causes are addressed

Control Charts

Common causes cannot be explained. This is the

inherent variation in a process. Only management can
take action to change the system.
Trying to adjust to common causes may increase
variation
Special causes need to be investigated and corrective
action taken

Common causes
Special causes
85 % of the problems are caused by the system
(management) and 15% by the workers

Demings causes of variation

Process Variation

(outputs of a process of shooting arrows)

Location and Dispersion

Reducing Fraction Defectives

Sequential Experiments and PDSA

Begin Module 2

End of Module 1

In order to be reasonably confident that a particular

change will bring about a desired/positive outcome,
the experiment needs to be carefully planned
Design of experiments is intended to provide a
framework for conducting the experiments in a
planned, methodical fashion
The results of the experiment are interpreted as a
prediction, i.e., how the system is expected to
behave in the future as a result of the changes
investigated through experimentation

In addition to special causes, there are common

causes of variation which require management
action to bring about improvement in the
system/process
Such action needs to be proactive where we attempt
to determine what changes will bring about positive
change
Experiments are needed to evaluate the effect of
making changes to the system so that improvement
can be effected.
7

Experiments

Proactive Action on a Process

12

11

10

What changes can we make that will

result in an improvement?

How will we know that a change is

an improvement?

Examples could be performance, time, usability,

aesthetics, durability, consistency, etc.

Need to clearly identify the criteria or measures

for improvement
Multiple measures that could be in conflict
Need to have a systematic way to prioritize
measures

How will we know that a change is an

improvement?

Customer complaints
Too much rework
Delays in delivery

Define objectives of the study

What are we trying to accomplish?

15

14

From Quality Improvement through Planned Experimentation by Ronal D.Moen,

Thomas W. Nolan, Lloyd P. Provost
13

Cycle for learning

and improvement

Current
Knowledge

What are we trying to accomplish?

The Model for Improvement

D
S

A
S

Testing a change

16

18

We need to clearly state how the test is to be performed,

i.e., who will conduct the test, where and when will it be
done, how many replications will be needed, what will be
outcome measures, etc.
We should look for ways to scale down and decrease the
time required for the initial test so that more knowledge is
gained quickly, which can be used in subsequent tests
Be innovative in the design of the test without getting too
complicated

During the Plan phase of the testing cycle:

17

Implementing a change

Testing Using PDSA Cycles

Developing a change

P
A

P
A
S

Sequential Building of Knowledge

Need to understand the nature of the problem at

hand
Need to be familiar with new technologies that
could help improve the process or system
Need to consider alternative materials and
alternative suppliers of the materials
Need to be innovative

What changes can we make that will result in

an improvement?

21

Is further testing required to increase the degree of belief?

Should we test other changes to study their effect?
Should modification be made to the current change and be
retested?
Should we alter the experiment conditions to retest the
effect of the change?
Are we ready to roll out the change on a full-scale basis?

During the Act phase of the testing cycle, we need to

determine what is the next course of action:

Testing Using PDSA Cycles

20

During the Study phase of the testing cycle, we compare what

our predictions were with regard to the outcome of the
change, to the actual results.
If the results do not match our prediction, we should attempt
to gain understanding why the prediction was not accurate
If the test outcome matches the prediction, then our degree
of belief in the change outcome is increased

Testing Using PDSA Cycles

19

Whatever be the outcome, we get information that helps us in

planning further tests and retests, if required

The test not being conducted properly

The support processes not being adequate
The result is actually not favorable

During the Do phase of the testing cycle, we actually conduct

the test or experiment. The outcome should provide data for
analysis and gaining knowledge about the effect of the
change.
If the outcome is not as expected, it might be due to:

Testing Using PDSA Cycles

It is important to design the test or experiment carefully as

the outcome and the interpretation of the results will
determine future course of actions
As discussed earlier, it is recommended that tests be
conducted in sequential cycles so that our degree of belief
increases that the change will result in improvement
It is also important to ensure that the improvement will be
sustained in a variety of circumstances
It is also essential to make sure that the outcome of the
experiment is due to the tested change as opposed to any
other reason

Testing the Change

Testing the Change

Begin Module 3

End of Module 2

24

23

22

Before the change

After the change

Delay

Delay

10

11

8
Week

10

11

Change made here

Alternate Scenario 1

Week

Change made here

12

12

13

13

FromQuality Improvement through Planned Experimentation

Delay

14

14

One of the methods to ascertain the effects of the

change is to conduct a measurement before (pre-test),
and a measurement after (post-test). For example
consider a change made in Week 7 and the results of the
pre-test in Week 4 and the post-test in Week 11 are given
below:

Pre-tests and Post-tests

27

26

25

Delay

Delay

Delay

10

11

10

11

8
Week

10

11

Change made here

Alternate Scenario 4

Week

Change made here

Alternate Scenario 3

Week

Change made here

12

12

12

13

13

13

14

14

14

Alternate Scenario 2

30

29

28

 End of Module 3

33

32

36

Are done to bring about a change in a product, process, or

system in the future
Since we are dealing with the future the universe is
dynamic
The aim of an analytic study is prediction that one of
several alternatives will be superior to the others in the
future
The role of the subject matter expert is very important in
analytic studies
Examples could be selection of one vendor from several
for the supply of a given part

Analytical studies

Enumerative and Analytical Studies

35

Are done with a universe that exists at present and static

The universe consists of tangible units, any number of
which may be sampled and studied
The aim of an enumerative study is estimation about some
aspect of the universe
Action may be taken on the universe based on the
estimate through sampling
Statisticians play a large role in enumerative studies
Examples of enumerative studies include Census, polling of
people, etc.

Enumerative studies

Test on a small scale and build knowledge

sequentially
Collect data over time
Include a wide range of conditions in the
sequence of tests

34

Enumerative and Analytical Studies

Design and Analysis of Experiments

Begin Module 4

Basic Principles for Testing Change

31

Plot a run chart with data from a few weeks

before and after the change
Conducting the test over a longer time period will
also add to our degree of belief
Remove the change and see if the process reverts
back to the performance before the change
Use planned experimentation

How can we make sure that the change in

output is caused by the change we made?

Testing the Change

 Experimental pattern the arrangement of factor

levels and experimental units in the design
Planned grouping forming blocks of experimental
units
Randomization the objective assignment of
combinations of factor levels to experimental units
Replication Repetition of experiments,
experimental units, measurements, etc.

Tools For Experimentation

39

38

Level A value or qualitative setting of a factor which would

be varied during an experiment
Experimental unit The smallest division of the experiment
that can receive a particular set of factor levels, e.g., a batch,
a single part, an hour of production, etc.
Blocks Groups of experimental units given similar treatment.
Usually, defined by background variables. The variation within
a block is expected to be less than the variation among blocks

Definitions

pt2

pt1

pt2

Head 2

pt2

41

Machine 1, Machine 2
Head 1, Head 2
part 1, part 2
m1, m2
42

m1 m2 m1 m2

pt1

Head 2

Each factor at two nested levels

The measurements are not in random order

Machine
Head within a machine
Part within a head
Measurement within a part

pt2

Machine 1

Pressure 2
Temp 1 Temp 2

m1 m2 m1 m2

pt1

Head 1

Nested Experimental Design

m1 m2 m1 m2

Four nested factors:

m1 m2 m1 m2

pt1

Head 1

Machine 1

Load 1
Batch 1 Load 2
Load 1
Batch 2 Load 2

Pressure 1
Temp 1 Temp 2

Full Factorial Design

40

The experimental pattern defines how the factors and factor

levels will be studied
There are a number of experimental patterns, such as
factorial, fractional factorial, and nested
In factorial design, each possible combination of factors is
examined. So an experiment with 4 factors, each at 2 levels,
will have 24 or 16 factor level combinations to be studied
In a fractional factorial design only a subset of the possible
combinations of factor levels are examined
In a nested design the levels of a factor are examined within a
given level of another factor

Response variable A variable observed or measured in an

experiment. It is the outcome from the experiment. An
experiment could have more than one response variables
Factor A variable deliberately changed in a controlled
manner to observe its impact on the response variable
Background variable A variable that can potentially affect a
response variable but is not a factor, e.g., time, lot, operator,
machine, etc.
Nuisance variable An unknown variable that can affect a
response variable. It is a background variable typically
unknown at the time of the experiment
37

Experimental Pattern

Definitions

43

45

Need to test material from 3 different suppliers. One factor at

3 levels
Five Background Variables
Background Variable
Levels
Machine
No. 6 and No. 10
Operator
Ram, Sunil, Gopal and Anil
Gauge
G-103 and G-233
Saw blade
10 blades available
Time (days)
Different days possible

Example of Chunk Variables

44

Suppose we want to test the wear on different brands of tires.

A natural grouping/blocking would be a car where 4 brands
could be tested on a car
Or if we wished to test the wear on the soles of shoes, a
natural grouping would be 2
When there are several background variables, we can
combine them to form chunks using some extreme values of
the background variables to test their effect on the
measurement of interest

Planned Grouping

We can either keep the background variables

constant
Or we can measure them
Or we can use planned grouping to set up blocks

How to control the background variables so

that the factor effects are not distorted
How to use background variables to establish
a wide range of conditions

Planned Grouping
No. 6
Ram
G-103
No. 1
Day 1

Block 2
B
C
A

No. 10
Sunil
G-233
No. 2
Day 2

Block 2

Block 3
C
A
B

No. 6
Gopal
G-103
No. 3
Day 3

Block 3

Block 4
A
C
B

No. 10
Anil
G-233
No. 4
Day 4

Block 4

46

End of Module 4

48

47

Repeated measurements of experimental units

Multiple experimental units for each combination of factors
Helps minimizing the effect of nuisance variables
Ideally it is better to get few replications on different days than get all
replications on the same day
No. of replication is constrained by the cost of the experiment

Replication

The assignment of factor combinations to experimental units and

sequence of testing is randomized
Helps reduce the variation due to nuisance variables from being
confused with variation due to factors

Randomization

Randomization and Replication

A, B, and C are the 3 suppliers whose samples have to be tested

Experiment grouping
Test
Block 1
1
B
2
A
3
C

Machine
Operator
Gauge
Saw blade
Day

Block 1

4 Blocks can be formed

23

Day 1
Pressure 30

1 2 3 4 5 6 7 8

Day 2
Pressure 80

1 2 3 4 5 6 7 8

Day 3
Pressure 20

1 2 3 4 5 6 7 8

Control Chart for Ranges

Control Chart for Averages

CL

UCL

LCL

UCL

50

Day 4
Pressure 40 51

1 2 3 4 5 6 7 8

The current clamp pressure is 30 pounds. It was decided to test

the dimension at 3 other clamp pressures, at 20 pounds, 40
pounds, and 80 pounds. Each of these would be tested for one
day. The control chart for the 4 days of experimentation are
shown next.

-7.3

10

7.3

49

The Xbar and R control charts for a critical dimension had been
maintained for 6 months. Recently tools were replaced and the
changes in averages were observed. It was suspected that it was
due to clamp pressure. So it was decided to test the effect of
clamp pressure on the variation in the dimension.

Example

Example of Experiment with a Single Factor

Begin Module 5

Pressure Average
lbs
Range
20
16
30
10
40
8
80
5

End of Module 5

We can conclude that clamp pressure reduces the average

range of the dimension. We should change the clamp pressure
To 80 lbs and monitor the situation to see if the effect is continued.
If the effect continues, we may wish to increase clamp pressure further

Day
3
1
4
2

Day 4: (pressure 40 pounds) Bothe charts in control. Slight decrease in

variation

Day 3: (pressure 20 pounds) Variation seems to have increased with one

point out of control Likely due to reduction in pressure

Day 2: (pressure 80 pounds) The Xbar chart is in control. In the R chart all
points below the CL. No other causes noted. Reduction in variation
likely due to increased clamp pressure

Day 1: (prior to experiment) Both Xbar and R charts are in control

Summary of control charts:

54

53

52