2.

Types of experiment
Good experimental design will help you to:
Improve the quality of your science
Get published in better journals
Save time and money
Use fewer animals

Types of experiment
Pilot experiments are small studies (1-20 experimental subjects) used to:

Test the logistics of a proposed larger study

Gain familiarity with the experimental material,
Ensure that treatments are not obviously excessively mild or severe
Check that staff are sufficiently well trained in the necessary procedures
Ensure that all steps in a proposed future experiment are feasible.
Gain some information on variability, although this will not usually be sufficiently
reliable to form the basis of power analysis calculations of sample size.

Exploratory experiments can be used to generate data with which to develop hypotheses for
future testing. They may work or not work. They may have no clearly stated hypothesis
(lets see what happens if.. is not a valid hypothesis on which to base an experiment).
Often they will measure many outcomes (characters). Picking out interesting looking
differences (known as data snooping) and then doing a hypothesis test to see if the
differences are statistically significant will lead to serious overestimation of the magnitude of
a response and excessive numbers of false positive results. Such differences should always
be tested in a controlled experiment where the hypothesis is stated a priori before the
results are published.
Depending on the nature of the data, statistical analysis will often be done using an analysis
of variance (ANOVA)
Confirmatory experiments are used to test some relatively simple hypothesis stated a priori.
This is the type of experiment mainly considered in this web site.
The basic principles are:

Experiments involve comparisons between two or more groups

Their aim is to test a null hypothesis that there is no difference among the groups
for the specified outcome.
If the null hypothesis is rejected at a certain level of probability (often 5%) this means
that the probability of getting a result as extreme as this or more extreme in the
absence of a true effect is 5% (assuming also that the experiment has been properly
conducted). So it is assumed that such a difference is likely to be the result of the
treatment. But, it could be a false positive resulting from sampling variation.
Failure to reject the null hypothesis does not mean that the treatment has no effect,
only that if there is a real effect this experiment failed to detect it. Absence of
evidence is not evidence of absence.

Experimental subjects need to be independently replicated because individuals (of

whatever type) vary. Two subjects can normally be regarded as being independent if
they can theoretically receive different treatments.
Subjects need to be assigned to groups, held in the animal house and measured at
random in order to minimise the chance of bias (a systematic difference between
groups)
As far as possible the experimenter should be blind with respect to the treatment
group in order to minimise bias.
The experiments need to be powerful, i.e. they should have a high probability of
detecting an effect of clinical or scientific importance if it is present.
In many cases a formal experimental design such as a completely randomised,
randomised block, Latin square etc. design will be used.
In most cases it is useful if the experiment has a wide range of applicability. In other
words the results should hold true under a range of different conditions (different
strains, both sexes, different diets, different environments etc.). At least some of
these factors should be explored using factorial and randomised block designs.

Experiments to explore relationships between variables. A typical example would be a

growth curve or a dose-response relationship. In these experiments the aim is often to test
whether the two variables are associated, and if so, what is the nature of that relationship.
The typical statistical analysis involves correlation and/or regression.

Research Strategy

L
r
w
e
w
u
u

T
m
o
2

H
u
i

C
D

More experiments needed

Types of experiments
By definition, all experiments involve manipulation of one or more
independent variables, and observing the effect on some outcome
(dependent variable). Experiments can be done in the field or in a
laboratory. They can involve human or animal subjects. What
distinguishes the type of experiment is the degree to which the
experimenter can assign subjects to conditions. Three types are
described here: True, Quasi- and Single-subject experiments.

True experiments
In a true experiment, subjects
are randomly assigned to the
treatment conditions (levels of
the independent variable). The
only differences in the groups
would be due to chance.
True experiments are excellent
for showing a cause-and-effect
relationship. Random
assignment (or random
assignment within matched
groups) controls for extraneous
variables.
They tend to be high
on internal validity. It is clear
what is being measured. There
still might be bias in the overall
research design, but at least
variables associated with
individuals are not a source
of constant error (see sources
of error in the Sampling
module).

Quasi-experiments

Quasi-experiments are sometimes

called natural experiments because
membership in the treatment level is
determined by conditions beyond the control of
the experimenter (subjects are already in the
box). An experiment may seem to be a true
experiment, but if the subjects have NOT been
randomly assigned to the treatment condition,
the experiment is a quasi- experiment
(quasi = seeming, resembles).
Experiments that take advantage of natural
occurrences are quasi-experiments, for
example, comparing achievement level of firstborn children with that of later-born children;
or comparing student performance at two
schools, one of which has a lower studentteacher ratio. The experimenter is unable to
assign subjects to treatment level - the
subjects are already in pre-existing groups.
One type of quasi experiment is to compare treatment versus
control conditions, where the assignment has occurred as a result
of some natural event. For example, comparing tranquilizer sales
in a community struck by a hurricane, with sales in a similar
community that was not affected by the hurricane. Or to compare
drinking levels at two colleges with similar student bodies, one
with an orientation programs with respect to alcohol use, and the
other having no such program. More....
Another type
of quasiexperiment is
to
compare pre-

Highway
fatalities before
and after
increasing the
speed limit

or behavior,
for example

Activism by
college students
before and after
an awareness
campaign

versus
post-events

Gun sales in a
community
Number of
before and
migratory
after a
cranes in
sensationalized
the
Sacramento killing
Valley
before and
after
wetland
habitat
restoration

Single-subject experiments
Instead of comparing behavior or performance of groups of people
at a single point in time, a single-subject experiment involves a
single case studied over a longer period of time. One individual or
situation is exposed to the varying levels of the independent
variable.
The most simple
single-subject
research design is
termed ABA, where A is the baseline (non-treatment or control)
condition or phase. B refers to the introduction of the treatment

factor. Behavior is recorded in both stages. Then there is a return

to A to see if in fact it was B that brought about the change. An
example might be treating a hyperactive child with a drug. Stage A
involves recording the child's behavior before any treatment, e.g.,
how many disruptive events in the classroom within a specified
period of time. Stage B would involve the same measurement after
the child has been treated. If B (the treatment) makes a
difference, returning to level A (no treatment) should result in a
return of the disruptive behavior. The basic research design can
include a second treatment phase -- ABAB, thereby increasing
the reliability and internal validity of the results.
The subject of a single-subject
experiment might be an entire
community. For example, police in a
small city introduced helicopter patrols
to see whether or not they resulted in
a decrease in burglaries. Here is a
figure showing the results.
This is an
ABABA
design.
The
occurrence
of
burglaries
tended to
be higher
when the
helicopter
was not
on patrol.
In setting
policy the
officials
would
have to
weigh the
benefit
against
the added
cost.

true vs. quasi-

and

lab vs. field studies

It is easy to confuse setting with experiment type. Use the table below to
keep them straight.
TYPE
LOCATION
Lab

True
common

Quasi- (natural)
unusual

Field

less
common

common

Although quasi-experiments are not commonly done in a lab, some of the

subject variables in lab-run experiments, such as gender or socioeconomic
level, are not subject to random assignment.

Take Self-test #3
Next section: Strengths and limitations