13-Steinberg-45460.

qxd

11/20/2007

5:04 PM

Page 147

13
Symbols: , Learning Objectives:

Errors and Significance

Terms: sampling error, significant difference, Type 1 error, alpha, Type 2 error, beta

Understand that probability, being uncertain, always includes error Distinguish between two types of errorType 1 and Type 2

Random Sampling Revisited

In Module 12, you learned how to state a hypothesis in terms of the expected effect of the independent variable on the dependent variable. You also learned how to sample subjects to be included in a study. When we sample randomly from a population, we expect the sample to reflect the characteristics of the population. That is what allows us to infer the characteristics of the larger population from the characteristics of the sample. Returning to the example of the effect of waters sweetness on the amount of water drunk, we expect that the difference we find in the amount of water drunk by subjects given sweetened or unsweetened water is about the same difference we would see in any subjects given sweetened or unsweetened water. We can then say that the difference we observe probably really exists in the populations from which the subjects were drawn that is, in the population of all subjects given sweetened water and all subjects not given sweetened water.

Sampling Error
Naturally, we do not expect sample statistics to exactly match the population parameters. To the degree that our sample statistics miss the true population parametersindicating that our subjects drank either more or less than what is true of the populationsany inference we make about the populations based on the samples will be in error. We call this type of error sampling error. Sampling error is any deviation due only to the particular cases falling within the samples. In other words, sampling error is deviation from expectation that is due to mere chance.
147

13-Steinberg-45460.qxd

11/20/2007

5:04 PM

Page 148

148

MODULE 13: ERRORS AND SIGNIFICANCE

Ordinarily, sampling error is small. If we toss a fair coin 100 times, it probably wont come up exactly 50 heads and 50 tails, but it will probably be close to thatsay, 46 heads and 54 tails. The difference from expectation is mere sampling error. Similarly, in the sweetened and unsweetened water example, we dont expect subjects in the two treatment conditions to drink exactly the same amount of water. But under the null hypothesis, we do expect them to drink similar amounts of water. Minor differences in the amount drunk are mere sampling error. Nevertheless, while sampling error is ordinarily small, it is possible to obtain a very large difference from expectation and yet still have it be due merely to sampling error. Its not likely, but it could happen. We will see that in a moment.

Significant Difference
How small does a difference from expectation have to be for it to be mere sampling error, and how big is too much for it to be mere sampling error? Lets return to the coin that we toss 100 times. Under the null hypothesis, we expect no difference in the number of heads and tails. In other words, we expect 50 heads and 50 tails. Now, suppose the coin comes up 97 heads and 3 tails. That certainly doesnt seem like a fair coin, does it? Such an outcome probably isnt just a chance occurrence. Rather, such a large deviation from the expected number of heads and tails would lead us to conclude that the coin is probably weighted. With such an outcome, we would reject the null hypothesis that the coin produces no difference in the number of heads and tails. This is the backward reasoningthe straw man processthat you met in Module 12. Similarly, in the study of the effect of sweetener on the amount of water drunk, the null hypothesis states that we expect no difference in the amount of water drunk by subjects given sweetened versus unsweetened water. Now, suppose we find a very large difference in the amount of water drunk by subjects given sweetened versus unsweetened water. As with the coin example, this outcome probably isnt just a chance occurrence. Rather, such a large difference in the amount of water drunk by subjects in the two treatment conditions would lead us to conclude that the presence of sweetener in the water does, indeed, make a difference in the amount of water drunk. With such an outcome we would reject the null hypothesis that there is no difference in the amount of water drunk by subjects given sweetened or unsweetened water. When we reject the null hypothesis, it is because we have found a large difference from expectation. In statistics, we call this a significant difference. A significant difference is any difference that is greater than would be expected by mere chance. If two comparable samples that differ only in their independent variable (IV) treatment show a significant difference in their dependent variable (DV) outcome, we can Statisticians do it assume that the difference in the DV outcome is due to the IV treatment. In significantly better. other words, the treatment probably caused the observed difference. Nevertheless, although large differences from expectation probably do reflect real treatment differences, it is possible to observe a very large difference from expectation and yet still have it be due merely to sampling error. It is not likely, but it could happen. We will see that in a moment.

CHECK YOURSELF! What is the relationship between sampling error and a significant difference?

13-Steinberg-45460.qxd

11/20/2007

5:04 PM

Page 149

MODULE 13: ERRORS AND SIGNIFICANCE

149

The Decision Table

Figure 13.1 shows a decision table. Perhaps you have seen one in a previous course in mathematics or logic. Across the top is the real truth. Its what is so in the entire population of cases, if only we had a way of knowing the real truthwhich we dont. Down the left side is what we think to be the real truth. Its the judgment we make about the real truth, based on the available sample data. Because we are making judgments about a population truth based on sample evidence, the decision table is a table of inference.
Real truth of the H0: in the population(s) True Your conclusion about the H0: based on the samples False Miss (Type 2) ()

You can find truth with logic only if you have already found truth without it. G. K. Chesterton

Retain

Hit

Reject

Miss (Type 1) ()

Hit

Figure 13.1

Elements of a Decision Table

And therein lies the rub. Because we are making inferences about cases beyond those actually included in the study, our decision may or may not agree with what is really true. Notice that two of the cells in the decision table are labeled Hit and two are labeled Miss. The Miss cells demonstrate that our decision might be wrong. Yes, thats right: We have left certainty behind and are now dealing with probabilityeducated guesses about what is so, but without certainty. Thats the way it will be for the rest of this textbook. Fortunately, however, probability is on our side.

Contradiction is not a sign of falsity, nor the lack of contradiction a sign of truth. Blaise Pascal

CHECK YOURSELF! In the following situations, have you made a hit or a miss? The null hypothesis is really true and your sample data leads you to retain it. The null hypothesis is really false and your sample data leads you to reject it.

Type 1 Error
Continuing with the drinking water example, our null hypothesis (H0) is that there is no difference in the amount of water drunk by subjects given sweetened water and those given unsweetened water. We can diagram the null hypothesis like this (see Figure 13.2).

13-Steinberg-45460.qxd

11/20/2007

5:04 PM

Page 150

150

MODULE 13: ERRORS AND SIGNIFICANCE

Amount of water drunk

A little

M or Unsweetened or Sweetened

A lot

Figure 13.2

Null Hypothesis for Amount of Water Drunk

NOTE: The solid line curve represents the amount of sweetened water drunk, and the dashed line curve represents the amount of unsweetened water drunk; larger curves represent populations, and smaller curves represent samples; is the population mean, and M is the sample mean.

Look again at the decision table. A true null hypothesis is located in the left column. Now, if the null hypothesis is really true, when we draw a random sample of subjects to whom we give sweetened water and we draw another random sample of subjects to whom we give unsweetened water, we probably will not But what is Truth? Is Truth find much difference in the amount of water each samples subjects unchanging law? We all drink. And if, indeed, we find very little difference between the samples have Truths. Are mine the same in the amount of fluid drunk, what would be the most logical decision as yours? to make regarding the null hypothesis? Should we retain it or reject it? Pontius Pilate, speaking to Yes, of course, we should retain (not reject) the H0. Jesus, in Webber and Rices rock Now, find the Retain decision point on the decision table. Its the opera, Jesus Christ, Superstar top row. And now, find where our decision (top row) intersects with the real truth (left column). They intersect at the top left cell. Notice that the top left cell says Hit. This indicates that we have made a correct decision regarding the real truth of the null hypothesis (see Figure 13.3).

Real truth of the H0: in the population(s) True Your conclusion about the H0: based on the samples False Miss (Type 2) ()

Retain

Hit

Reject

Miss (Type 1) ()

Hit

Figure 13.3

Correct Decision Under a True Null Hypothesis

13-Steinberg-45460.qxd

11/20/2007

5:04 PM

Page 151

MODULE 13: ERRORS AND SIGNIFICANCE

151

Given that the null hypothesis is really true, most of the time we will not reject the null hypothesis. Most of the time, we will make a hit. But does it have to happen that way? No, it does not. It is possible, even when the null hypothesis is really true, and even when subjects are randomly sampled, that we will find a large difference in the amount of water drunk by the two groups of subjects. Its not likely, but it could happenin the same way that the fair coin we examined in Modules 10 and 11 could come up heads all the time or none of the time. We can diagram the situation like this (see Figure 13.4).
Amount of water drunk

A little

M Unsweetened

Unsweetened or Sweetened

M Sweetened

A lot

Figure 13.4

Disparate Samples Under a True Null Hypothesis

NOTE: The solid line curve represents the amount of sweetened water drunk, and the dashed line curve represents the amount of unsweetened water drunk; larger curves represent populations, and smaller curves represent samples; is the population mean, and M is the sample mean.

But remember, we never know the real truth regarding the null hypothesis. The only data available to us come from the subjects in our samples. So if we find a large difference between the samples in the amount of water drunk, as shown in Figure 13.4, what would be the most logical decision to make regarding the null hypothesis? Should we retain it or reject it? Yes, of course, we should reject the null hypothesis. Now, find the Reject decision point on the decision table (see Figure 13.5). Its the bottom row. And find where our decision (bottom row) intersects with the real truth (left column).

Real truth of the H0: in the population(s) True Your conclusion about the H0: based on the samples data False Miss (Type 2) ()

Retain

Hit

Reject

Miss (Type 1) ()

Hit

Figure 13.5

Decision Table Showing Type 1 Error

13-Steinberg-45460.qxd

11/20/2007

5:04 PM

Page 152

152

MODULE 13: ERRORS AND SIGNIFICANCE

They intersect at the bottom left cell. Notice that the bottom left cell says Miss. This indicates that we have made an incorrect decision regarding the real truth of the null hypothesis. There is a name for this type of error. Its called a Type 1 error. It is also called alpha and is designated by the symbol . Type 1 error, or , occurs when the null hypothesis is really true, but we incorrectly reject it. Said another way, its when there really is no effect, but we find one.

Type 2 Error
So far, we have looked only at the situation when the null hypothesis is really true. But what if the research hypothesis, rather than the null hypothesis, is really true? Continuing with the drinking water example, the nondirectional research hypothesis (HA) is that there really is a difference in the amount of water drunk by subjects given sweetened water versus unsweetened water. We can diagram the research hypothesis like this (see Figure 13.6).
Amount of water drunk

A little

M or Unsweetened

M or Sweetened

A lot

Figure 13.6

False Null Hypothesis for Amount of Water Drunk

NOTE: The solid line curve represents the amount of sweetened water drunk, and the dashed line curve represents the amount of unsweetened water drunk; larger curves represent populations, and smaller curves represent samples; is the population mean, and M is the sample mean.

Look again at the decision table. A true research hypothesis is the same as a false null hypothesis. Therefore, the research hypothesis is located in the right column. Now, if the null hypothesis is really false, then when we draw a random sample of subjects to whom we give sweetened water and we draw another random sample of subjects to whom we give unsweetened water, probably we will find quite a bit of difference in the amount of water each samples subjects drink. And if, indeed, we do find quite a lot of difference between the samples in the amount of water drunk, what would be the most logical decision to make regarding the null It is not certain that hypothesis? Should we retain it or reject it? Yes, of course, we should everything is uncertain. reject the H0. Blaise Pascal Now, find the Reject decision point on the decision table. Its the bottom row. And now, find where our decision (bottom row) intersects with the real truth (right column). They intersect at the bottom right cell. Notice that the bottom right cell says Hit. We have made a correct decision regarding the real truth of the null hypothesis (see Figure 13.7).

13-Steinberg-45460.qxd

11/20/2007

5:04 PM

Page 153

MODULE 13: ERRORS AND SIGNIFICANCE

Real truth of the H0: in the population(s) True Your conclusion about the H0: based on the samples data False Miss (Type 2) ()

153

Retain

Hit

Reject

Miss (Type 1) ()

Hit

Figure 13.7

Correct Decision Under a False Null Hypothesis

Given that the null hypothesis is really false, rejecting the null hypothesis is the most likely outcome. Most of the time, we will make a hit. But does it have to happen that way? No, it does not. It is possible, even when the null hypothesis is really false, and even when subjects are randomly sampled, that we would not find much of a difference in the amount of water drunk by the two groups. Its not likely, but it could happenin the same way that a weighted coin could come up heads half the time and tails the other half of the time. We can diagram the situation like this (see Figure 13.8).
Amount of water drunk

A little

Unsweetened

M Unsweetened or Sweetened

Sweetened

A lot

Figure 13.8

Homogeneous Samples Under a False Null Hypothesis

NOTE: The solid line curve represents the amount of sweetened water drunk, and the dashed line curve represents the amount of unsweetened water drunk; larger curves represent populations, and smaller curves represent samples; is the population mean, and M is the sample mean.

But remember, we never know the real truth regarding the null hypothesis. The only data available to us come from the subjects in our samples. So if we find very little difference between the samples in the amount of water drunk, as shown in Figure 13.8, The search for truth is what would be the most logical decision to make regarding the null hypothemore precious than its sis? Should we retain it or reject it? Yes, of course, we should retain (fail to possession. reject) the null hypothesis. Now, find the Retain decision point in the decision table. Its the top Albert Einstein row. And find where our decision (top row) intersects with the real truth

13-Steinberg-45460.qxd

11/20/2007

5:04 PM

Page 154

154

MODULE 13: ERRORS AND SIGNIFICANCE

(right column). They intersect at the top right cell. Notice that the top right cell says Miss. This indicates that we have made an incorrect decision regarding the real truth of the null hypothesis (see Figure 13.9).
Real truth of the H0: in the population(s) True Your conclusion about the H0: based on the samples data False Miss (Type 2) ()

Retain

Hit

Reject

Miss (Type 1) ()

Hit

Figure 13.9

Decision Table Showing Type 2 Error

There is a name for this type of error. Its called Type 2 error. It is also called beta and is designated by the symbol . Type 2 error, or , occurs when the null hypothesis is really false, but we incorrectly retain it. Said another way, its when there really is an effect, but we dont find one. Because hypothesis testing always involves the possibility of error, we will want to test our hypotheses with the least amount of error. Therefore, consideration of Type 1 and Type 2 errors underlies everything we will do for the remainder of this textbook.

CHECK YOURSELF! Draw a decision table like the one in this module but with the two column headings reversed. Keep the two row headings where they were. Now, enter hit and miss designations in the cells according to the definitions in this module. In what cell does each designation fall? Are they the same cells as in this module? Why or why not? Draw another decision table, this time leaving the two column headings as they were in this module but reversing the two row headings. Then, enter hit and miss designations in the cells according to the definitions in this module. In what cell does each designation fall? Are they the same cells as in this module? Why or why not?

PRACTICE
In answering the following exercises, it will be helpful to have before you the definitions of Type 1 and Type 2 error that you have already learned. Here is a reminder: Type 1: There really is not an effect (population truth), but you find one (your decision) in your study. Type 2: There really is an effect (population truth), but you dont find one (your decision) in your study.

13-Steinberg-45460.qxd

11/20/2007

5:04 PM

Page 155

MODULE 13: ERRORS AND SIGNIFICANCE

155

1. A man is accused of committing a crime. The jury finds the man not guilty. Create a decision table for this situation like the one in this module and answer the following questions: a. In which row of the table does the jurys action fall? b. Assume that the man really did commit the crime. In which column of the table does his guilt or innocence fall? c. By finding this guilty man not guilty, which type of error did the jury makeType 1 or Type 2? 2. It is a population fact that men and women differ in height. You draw a random sample of men and a random sample of women, measure their heights, and find that the average height of the men and women do not differ. Given that mens and womens heights really are different in their respective populations, what type of error occurred in your studyType 1 or Type 2? Explain your answer. 3. People who have infectious mononucleosis (mono) test positive for the virus on a heterophile test of sheep cell agglutination. A man who does not have mono but is merely tired and nursing a cold takes the diagnostic test, and the test result is positive. What type of error is this false positiveis it Type 1 or Type 2? Explain your answer. 4. Stan applied to work at Great Company. Because Stan had a degree from a prestigious university, the company expected him to be a great employee. Given their expectation (i.e., given what they thought was the population truth regarding applicants from prestigious universities), managers at Great Company hired Stan. However, Stan turned out to be a very unproductive worker, and Great Company ended up firing him. What type of error did Great Company make in its initial decision to hire Stanwas it Type 1 or Type 2? Explain your answer. 5. Two fishermen, Abe and Gabe, are debating the relative merits of two lakes for fishing. Based on their experience in fishing in the two lakes, they decide that the two lakes are equivalent in terms of the number of fish. A researcher overhears their conversation and decides to test their conclusion. Using advanced sonar sensing, she counts the actual number of fish in each lake. She finds that one lake has many more fish than the other lake does. Assuming that her findings reflect the population truth about the two lakes, what type of error did Abe and Gabe make in their judgmentwas it Type 1 or Type 2? Explain your answer.

Visit the study site at www.sagepub.com/steinbergsastudy for practice quizzes and other study resources.