Chap 8 & 9.

Gathering data: Observational Studies and Experiments

Contents
1

Three studies on cell phone usage and cancer

The difference between observational studies and experiments 3 Observational study

3.1 3.2 3.3 3.4

5
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 7

Populations and samples Kinds of sample designs . Other sampling designs

. . . . . . . . . . . . . . . . . . . . . . 13 . . . . . . . . . 14

Other possible sources of bias in surveys

Experiments
4.1 4.2 4.3 4.4 4.5 4.6

16
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 . . . . . 18

Terminologies

The importance of comparison in experiments Example: the Lung Health Study Randomization . Replication . . . .

. . . . . . . . . . . . . . . 22

. . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

Summary of the logic of randomized comparative experiments . . . . . . . . . . . 28

1

1 Three studies on cell phone usage and cancer

Study 1. A german study in 2001 compared 118 patients with a rare form of eye cancer to 475 healthy patients who did not have the eye cancer. The patients cell phone use was measured using a questionnaire. The eye cancer patients used cell phones more often, on the average. Study 2. A US study in 2000 compared 469 patients with brain cancer to 433 patients who did not have brain cancer. The patients cell phone use was measured using a questionnaire. The two groups use of cell phones was similar. Study 3. An Australian study in 1997 conducted an experiment with 200 mice. 100 mice were exposed for 2.5 hrs a day to the same kind of microwaves with roughly the same power as the kind transmitted from a cell phone. The other 100 mice were not exposed. After 18 month, the brain tumor rate for the mice exposed to cell phone radiation was twice as high as the brain tumor rate for the unexposed mice.

2 The difference between observational studies and exper-

iments

In an experiment, the investigator studies the effect of varying some factor that he/she controls. In an observational study, the investigator merely observes and records information on the subjects but does not manipulate any factors. For both ways to gather data, it is very difcult to establish causation between one variable and another: there is always the possibility that some lurking variable could be responsible for the association. Establishing cause and effect is especially difcult based on observational studies. It is easier to adjust for lurking variables in an experiment than in an observational study. We can study the effect of an explanatory variable on a response variable more accurately with an experiment than with an observational study. When our goal is to understand cause and effect, experiments are the only source of convincing data.

Practice: are there possible lurking variables in the following observational studies?

A study suggest that heavier user of cell phone have higher chance of getting eye cancer. Some observational studies suggest that drinking wine rather than beer or spirits confers added health benets.

Question: if experiment is preferable, why ever conduct an observational study?

unethical, hard to apply treatments correctly, time span of study; not all studies are concerned with cause and effect.

3 Observational study

3.1

Populations and samples

A population is the entire set of items about which we might wish to draw conclusions. Example: I wish to nd out the average income of families of current UI undergrads. Example: A political pollster would like to know the Presidential preference of every registered voter in Florida. Some populations we would like to study are hypothetical. Example: all pregnant women who are infected with the HIV virus now and in the future A sample is the subset of the population that we can actually study (on which we can measure values of variables). How a sample is drawn from a population affects how valid it is to apply conclusions based on the sample to the population. The sample design is the method used to choose the sample from the population.

Bias The results of a study are biased if they are subject to systematic error. i.e., there is something about the way the study is carried out such that, if we did many studies in this way, on average wed get the wrong conclusions! One source of bias is if the sample is not representative of the entire population. The design of a study is biased if it systematically favors certain outcomes.

3.2

Kinds of sample designs

simple random sample (SRS) a sample of size n individuals chosen in such a way that every set of n indivduals in the population has an equal chance to be the sample the ideal biased or unbiased? voluntary response sample consists of people who choose themselves by responding to a general appeal biased or unbiased? convenience sample consists of subjects who are easy to get biased or unbiased? judgment sample consists of subjects chosen by an expert to be representative of the population biased or unbiased?

How simple random samples are drawn each member of the population is uniquely identied in some way example: the population of interest is UI students; each has a unique ID number intuitive idea: the identiers are put in a hat and drawn at random

in old days, done manually using a table of random digits rst assign a unique numeric label to each member of the population use table of digits (eg. Table B in your book) to select labels at random. nowadays, usually done by a computer

Example I wish to get an idea as to how well students in 22S:105 like the textbook. To do this, I want to administer a lengthy interview and I have time to do only 3. Therefore, I want to draw a simple random sample of size 3 from the population of 11 students registered for 22S:105. Let us use SAS to draw a simple random sample options linesize = 79 ; data students ; input name $9. ; datalines ; Marija Matthew H Joseph Lingzi Sangeeta Kevin Frances Cheryl Eric Qianhao Matthew Z ;

proc print run ;

Output:

data = students ;

Obs 1 2 3 4 5 6 7 8 9 10 11

Name Marija Matthew H Joseph Lingzi Sangeeta Kevin Frances Cheryl Eric Qianhao Matthew Z

proc plan seed = 123 ; factors a = 3 of 11 ; run ;

Output: The PLAN Procedure Select Levels 3 ----a--9
10

Factor a

Order Random

11

Using the same seed will reproduce exactly the same random choice! proc plan seed = 123 ; factors a = 3 of 11 ; run ; Factor a The PLAN Procedure Select Levels 3 ----a--9 5 4 11 Order Random

Using a different seed will produce a different set of choices. proc plan seed = 3 ; factors a = 3 of 11 ; run ; Factor a The PLAN Procedure Select Levels 3 ----a--7 11 4 11 Order Random

11

Drawing from a larger population proc plan seed = 241 ; factors a = 100 of 1000 ; run ;
Procedure PLAN Factor -----a Select -----100 Levels -----1000 Order ------Random

a ----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+ 576 630 550 119 901 497 864 792 705 120 944 767 773 481 359 286 441 692 507 687 362 517 412 921 265 819 449 584 110 597 139 432 844 41 28 598 868 644 470 518 424 479 859 488 269 311 264 24 594 144 621 861 585 822 326 235 9 240 775 69 897 863 337 52 674 529 329 271 178 175 462 651 168 143 820 752 262 923 939 562 239 423 673 298 50 974 435 233 202 419 281 996 341 834 648 865 57 904 442 358

12

3.3

Other sampling designs

A probability sample gives each member of the population of interest a known chance of being selected. stratied random sampling procedure rst divide the population into strata groups of similar individuals draw a simple random sample from each stratum combine the SRSs to form the full sample ensures that each stratum is represented in the overall sample Example: in the survey of class opinions on the textbook, I might divide the class into men and women and take a SRS within each gender Example: IRS uses stratied samples during audits to verify a persons tax return based on dollar amounts. Probability sampling methods other than SRSs require more complicated statistical analysis than do SRSs. But meaningful results can be obtained because we know what population was actually sampled and exactly how it was done. This contrasts with voluntary response samples, convenience samples, and judgment samples.

13

3.4

Other possible sources of bias in surveys

1. Undercoverage The list of individual items from which a sample is chosen is called the sampling frame Some segments of the population of interest are likely to be missed even with careful sampling methods because they are not included in the sampling frame Example: telephone surveys systematically miss the 6% of American households without phones. 2. Nonresponse Some members of the chosen sample cannot be contacted or refuse to answer. This biases the results of the survey if the members who do not respond are different from the general population. Example: in surveys that include questions about household income, families with unusually low or unusually high incomes are less likely to answer that question than are families with moderate income.

14

3. Response bias Respondents may lie, especially about sensitive subjects. Attributes or behavior of interviewers can make this more likely. Example: In a survey concerning roles of family members, a father might tend to respond differently to the question How many hours per week do you spend caring for your children on average? depending on the gender of the interviewer. 4. Bias due to wording of questions leading questions questions involving undened terms Example: Do you eat 5 servings of fruits and vegetables per day?

15

4 Experiments

4.1

Terminologies

Recall: What is the critical difference between an experiment and an observational study?

experimental units: individual items on which experiment is done usually called subjects when they are human we can measure a response variable individually on each experimental unit treatment: a specic experimental condition, controlled by the experimenter, and applied to the units

16

 example: agricultural eld experiment land available for use in the experiment is divided up into equal-sized plots; each plot is an experimental unit same variety of corn planted in all plots response variable for each plot is average number of bushels of corn harvested per acre treatments are different types of fertilizers assigned to plots factor: a particular explanatory variable manipulated by the experimenter a factor has one or more levels different values that are assigned to different units e.g., each type of fertilizer in the agricultural example is a different level of the factor fertilizer type A single experiment may involve more than one factor. In this case, each treatment is dened as the combination of levels of different factors. example: more complex agricultural eld experiment factor A: fertilizer type with 3 levels factor B: variety of corn, with 2 levels then one of 6 possible treatments is assigned to each plot

17

4.2

The importance of comparison in experiments

Comparative experiments are used to separate the effects of an experimental treatment from those of extraneous variables. important when we cant control all extraneous variables

Example: A college regularly offers a review course to prepare candidates for the Graduate Management Admission test (GMAT). This year, it offers only an online version of the course. The average GMAT score of students in the online course in 10% higher than the longtime average for those who took the classroom review course. Is the online course more effective? Cant tell! We have no way of knowing what other variables might have inuenced the GMAT score. Solution: we need to do a comparative experiment.

18

Comparative experiment Groups of subjects in a comparative experiment: experimental group(s) receive treatment(s) the effects of which are under study control group receives no treatment or a sham treatment

Example: study reported in Consumer Reports, Feb. 1976 a group of senior citizens was randomly divided into 2 groups group 1: daily doses of vitamin C group 2: no treatment At end of winter, vitamin C group reported fewer colds than no-treatment group. Investigator concluded that vitamin C helps to prevent colds.

19

Two problems unique to the experimental designs: 1) The placebo effect denition: A placebo is a dummy treatment no direct (physical) effect on response variable In another study decribed in the same Consumer Reports article, two treatment groups one group of subjects were given daily vitamin C and told it was a placebo other group received a placebo and were told it was vitamin C The group who thought they were receiving vitamin C reported fewer colds. An aside concerning medical studies Note: A study of the last-mentioned type would be considered unethical today. informed consent required for participation in clinical trials For testing new treatments of diseases or conditions for which a treatment already exists, the best standard treatment is given to the control group. It would be considered unethical to withhold an effective known treatment
20

2) Blinding in experiments with human subjects refers to preventing some people involved in the experiment from knowing which subjects are receiving which treatment single-blind experiment: subjects do not know which treatment they are receiving, but study personnel are not blinded double-blind experiment: neither the subjects nor any study personnel who administer treatment or evaluate response variable know which treatment subjects are receiving

21

4.3

Example: the Lung Health Study

clinical trial sponsored by the NIH involving 10 clinical centers in the US and Canada aim: to determine the effects on the decline of lung function in smokers already at risk for COPD (a lung disease) of: a stop smoking program daily use of an inhaled asthma drug response variable: change in FEV1 ( a measure of lung function ) from the time a subject entered the study until a follow-up visit 5 years later subjects: approximately 6000 smokers with mild impairment of lung function groups Usual Care group (control group) received neither the smoking cessation program nor any medication Special Intervention Placebo group received the smoking cessation program but a placebo inhaler Special Intervention Active drug group received the smoking cessation program and the active inhaled drug

22

 Was the LHS an experiment or an observational study? Was it comparative? What were the factors? What were the treatments?

Blinding in the LHS Patients and study personnel knew who was in UC group. Patients and study personnel knew everyone in both SI groups received smoking cessation program. Double blinding as to which SI patients were receiving placebo and which active drug. Neither patients, clinic personnel, nor study directors knew this until end of study.

23

4.4

Randomization

Another aspect of experimental design is how to determine which experimental units receive which treatment. Randomization refers to assignment of experimental units to treatments based on chance purpose: effort to make sure the experimental groups are not systematically different from one another in ways other than the treatment assignment in particular, subjects are not assigned by the experimenter carried out by computers

24

Here are three popular randomized designs.

1. Completely randomized design All the experimental units are allocated at random among all the treatments. example: if LHS had had a completely randomized design, idea would have been: put 6000 envelopes in a hat, each with a slip of paper inside, 2000 saying UC, 2000 saying SIP and 2000 saying SIA each time a new patient enrolls in the study, draw an envelope at random and have the pharmacists dispense the appropriate treatment

2. matched pairs design can be used only if there are only 2 treatments subjects are paired up, so each pair is as similar as possible on important known factors that might affect the response variable for each pair, randomly as one of the treatments to each subject
25

3. randomized block design block: a group of experimental units that are known before the experiment to be similar in some way that may affect the response variable randomized block design: randomization of units to treatments is carried out separately within each block in LHS, the patients enrolled by each of the 10 different clinics were a block randomization to the treatments was carried out separately within each clinics patients to make sure all treatment groups were represented within each clinic Why?

26

4.5

Replication

Imagine that the Lung Health Study had had only 2 patients in each treatment group instead of 2000. Experiments need to use a large enough number of experimental units to reduce chance variation to within acceptable bounds. As we study different methods of statistical analysis, we will learn how to compute sample sizes. An observed effect so large that it would rarely occur by chance is called statistically signicant. We will use the laws of probability to learn how likely we would be to see treatment effects as large as those observed by pure chance.

27

4.6

Summary of the logic of randomized comparative experiments

Our goal is to obtain good evidence that differences in the treatments actually cause the differences we see in the response. Randomization forms experimental groups that are likely to be similar in all respects except treatment assignment. Comparative design ensures that inuences other than the experimental treatments operate equally on all groups. Consequently, differences between treatment groups in average response variable must be due to either effects of treatment pure chance*

Use more subjects in each group can reduce the difference in response between different treatment groups due to pure chance.

An observed effect so large that it would rarely occur by chance is called statistically signicant.

28