RESEARCH METHODS TESTS FOR SIGNIFICANCE What are Tests for Significance Steps in Testing for Statistical Significance

1) State the Research Hypothesis 2) State the Null Hypothesis 3) Type I and Type II Errors Select a probability of error level (alpha level) 4) Chi Square Test Calculate Chi Square Degrees of freedom Distribution Tables Interpret the results 5) T-Test Calculate T-Test Degrees of freedom Distribution Tables Interpret the results Reporting Tests of Statistical Significance Final Comments

What Are Tests for Significance Two questions arise about any hypothesized relationship between two variables: 1) what is the probability that the relationship exists; 2) if it does, how strong is the relationship There are two types of tools that are used to address these questions: the first is addressed by tests for statistical significance; and the second is addressed by Measures of Association. Tests for statistical significance are used to address the question: what is the probability that what we think is a relationship between two variables is really just a chance occurrence? If we selected many samples from the same population, would we still find the same relationship between these two variables in every sample? If we could do a census of the population, would we also find that this relationship exists in the population from which the sample was drawn? Or is our finding due only to random chance? Tests for statistical significance tell us what the probability is that the relationship we think we have found is due only to random chance. They tell us what the probability is that we would be making an error if we assume that we have found that a relationship exists. We can never be completely 100% certain that a relationship exists between two variables. There are too many sources of error to be controlled, for example, sampling error, researcher bias, problems with reliability and validity, simple mistakes, etc.

But using probability theory and the normal curve, we can estimate the probability of being wrong, if we assume that our finding a relationship is true. If the probability of being wrong is small, then we say that our observation of the relationship is a statistically significant finding. Statistical significance means that there is a good chance that we are right in finding that a relationship exists between two variables. But statistical significance is not the same as practical significance. We can have a statistically significant finding, but the implications of that finding may have no practical application. The researcher must always examine both the statistical and the practical significance of any research finding. For example, we may find that there is a statistically significant relationship between a citizen's age and satisfaction with city recreation services. It may be that older citizens are 5% less satisfied than younger citizens with city recreation services. But is 5% a large enough difference to be concerned about? Often times, when differences are small but statistically significant, it is due to a very large sample size; in a sample of a smaller size, the differences would not be enough to be statistically significant.

Steps in Testing for Statistical Significance a. b. c. d. e. State the Research Hypothesis State the Null Hypothesis Select a probability of error level (alpha level) Select and compute the test for statistical significance Interpret the results

1) State the Research Hypothesis A research hypothesis states the expected relationship between two variables. It may be stated in general terms, or it may include dimensions of direction and magnitude. For example, General: The length of the job training program is related to the rate of job placement of trainees. Direction: The longer the training program, the higher the rate of job placement of trainees. Magnitude: Longer training programs will place twice as many trainees into jobs as shorter programs. General: Graduate Assistant pay is influenced by gender. Direction: Male graduate assistants are paid more than female graduate assistants. Magnitude: Female graduate assistants are paid less than 75% of what male graduate assistants are paid. 2) State the Null Hypothesis A null hypothesis usually states that there is no relationship between the two variables. For example,

There is no relationship between the length of the job training program and the rate of job placement of trainees. Graduate assistant pay is not influenced by gender. A null hypothesis may also state that the relationship proposed in the research hypothesis is not true. For example, Longer training programs will place the same number or fewer trainees into jobs as shorter programs. Female graduate assistants are paid at least 75% or more of what male graduate assistants are paid. Researchers use a null hypothesis in research because it is easier to disprove a null hypothesis than it is to prove a research hypothesis. The null hypothesis is the researcher's "straw man." That is, it is easier to show that something is false once than to show that something is always true. It is easier to find disconfirming evidence against the null hypothesis than to find confirming evidence for the research hypothesis.

3) TYPE I AND TYPE II ERRORS Even in the best research project, there is always a possibility (hopefully a small one) that the researcher will make a mistake regarding the relationship between the two variables. There are two possible mistakes or errors. The first is called a Type I error. This occurs when the researcher assumes that a relationship exists when in fact the evidence is that it does not. In a Type I error, the researcher should accept the null hypothesis and reject the research hypothesis, but the opposite occurs. The probability of committing a Type I error is called alpha. The second is called a Type II error. This occurs when the researcher assumes that a relationship does not exist when in fact the evidence is that it does. In a Type II error, the researcher should reject the null hypothesis and accept the research hypothesis, but the opposite occurs. The probability of committing a Type II error is called beta. Generally, reducing the possibility of committing a Type I error increases the possibility of committing a Type II error and vice versa, reducing the possibility of committing a Type II error increases the possibility of committing a Type I error. Researchers generally try to minimize Type I errors, because when a researcher assumes a relationship exists when one really does not, things may be worse off than before. In Type II errors, the researcher misses an opportunity to confirm that a relationship exists, but is no worse off than before. In this example, which type of error would you prefer to commit? Research Hypothesis: El Nino has reduced crop yields in County X, making it eligible for government disaster relief. Null Hypothesis: El Nino has not reduced crop yields in County X, making it ineligible for government disaster relief.

If a Type I error is committed, then the County is assumed to be eligible for disaster relief, when it really is not (the null hypothesis should be accepted, but it is rejected). The government may be spending disaster relief funds when it should not, and taxes may be raised. If a Type II error is committed, then the County is assumed to be ineligible for disaster relief, when it really is eligible (the null hypothesis should be accepted, but it is rejected). The government may not spend disaster relief funds when it should, and farmers may go into bankruptcy. In this example, which type of error would you prefer to commit? Research Hypothesis: The new drug is better at treating heart attacks than the old drug Null Hypothesis: The new drug is no better at treating heart attacks than the old drug If a Type I error is committed, then the new drug is assumed to be better when it really is not (the null hypothesis should be accepted, but it is rejected). People may be treated with the new drug, when they would have been better off with the old one. If a Type II error is committed, then the new drug is assumed to be no better when it really is better (the null hypothesis should be rejected, but it is accepted). People may not be treated with the new drug, although they would be better off than with the old one.

SELECT A PROBABILITY OF ERROR LEVEL (ALPHA LEVEL) Researchers generally specify the probability of committing a Type I error that they are willing to accept, i.e., the value of alpha. In the social sciences, most researchers select an alpha=.05. This means that they are willing to accept a probability of 5% of making a Type I error, of assuming a relationship between two variables exists when it really does not. In research involving public health, however, an alpha of .01 is not unusual. Researchers do not want to have a probability of being wrong more than 0.1% of the time, or one time in a thousand. If the relationship between the two variables is strong (as assessed by a Measure of Association), and the level chosen for alpha is .05, then moderate or small sample sizes will detect it. As relationships get weaker, however, and/or as the level of alpha gets smaller, larger sample sizes will be needed for the research to reach statistical significance.

4) The Chi Square Test For nominal and ordinal data, Chi Square is used as a test for statistical significance. For example, we hypothesize that there is a relationship between the type of training program attended and the job placement success of trainees. We gather the following data:

Type of Training Attended: Vocational Education Work Skills Training

Number attending Training 200 250

Total

450

Placed in a Job? Yes No Total

Number of Trainees 300 150 450

To compute Chi Square, a table showing the joint distribution of the two variables is needed: Table 1. Job Placement by Type of Training (Observed Frequencies)

Placed in a Job?

Yes No Total

Type of Training Vocational Education 175 25 200

Work Training 125 125 250

Skills Total 300 150 450

Chi Square is computed by looking at the different parts of the table. The "cells" of the table are the squares in the middle of the table containing numbers that are completely enclosed. The cells contain the frequencies that occur in the joint distribution of the two variables. The frequencies that we actually find in the data are called the "observed" frequencies. In this table, the cells contain the frequencies for vocational education trainees who got a job (n=175) and who didn't get a job (n=25), and the frequencies for work skills trainees who got a job (n=125) and who didn't get a job (n=125). The "Total" columns and rows of the table show the marginal frequencies. The marginal frequencies are the frequencies that we would find if we looked at each variable separately by itself. For example, we can see in the "Total" column that there were 300 people who got a job and 150 people who didn't. We can see in the "Total" row that there were 200 people in vocational education training and 250 people in job skills training. Finally, there is the total number of observations in the whole table, called N. In this table, N=450.

Calculate Chi Square 1) display observed frequencies for each cell 2) calculate expected frequencies for each cell 3) calculate, for each cell, the expected minus observed frequency squared, divided by the

expected 4) all up the results for all the cells

frequency

To find the value of Chi Square, we first assume that there is no relationship between the type of training program attended and whether the trainee was placed in a job. If we look at the column total, we can see that 300 of 450 people found a job, or 66.7% of the total people in training found a job. We can also see that 150 of 450 people did not find a job, or 33.3% of the total people in training did not find a job. If there was no relationship between the type of program attended and success in finding a job, then we would expect 66.7% of trainees of both types of training programs to get a job, and 33.3% of both types of training programs to not get a job. The first thing that Chi Square does is to calculate "expected" frequencies for each cell. The expected frequency is the frequency that we would have expected to appear in each cell if there was no relationship between type of training program and job placement. The way to calculate the expected cell frequency is to multiply the column total for that cell, by the row total for that cell, and divide by the total number of observations for the whole table. For the upper left hand corner cell, multiply 200 by 300 and divide by 450=133.3 For the lower left hand corner cell, multiply 200 by 150 and divide by 450=66.7 For the upper right hand corner cell, multiply 250 by 300 and divide by 450=166.7 For the lower right hand corner cell, multiply 250 by 150 and divide by 450=83.3 Table 2. Job Placement by Type of Training (Expected Frequencies)

Placed in a Job?

Yes No Total

Type of Training Vocational Education 133.3 66.7 200

Work Training 166.7 83.3 250

Skills Total 300 150 450

This table shows the distribution of "expected" frequencies, that is, the cell frequencies we would expect to find if there was no relationship between type of training and job placement. Note that Chi Square is not reliable if any cell in the contingency table has an expected frequency of less than 5. To calculate Chi Square, we need to compare the original, observed frequencies with the new, expected frequencies. For each cell, we perform the following calculations: a) Subtract the value of the observed frequency from the value of the expected frequency

b) square c) divide the result by the value of the expected frequency For each

the

result

cell

above,

fe - fo
(133.3 - 175) (66.7 - 25) (166.7 - 125) (83.3 - 125)

(fe - fo)2
(133.3 - 175) (66.7 - 25)2 (166.7 - 125)2 (83.3 - 125)2
2

[(fe - fo)2] / fe
[(133.3 - 175) ] / 133.3 [(66.7 - 25)2] / 66.7 [(166.7 - 125)2] / 166.7 [(83.3 - 135)2] / 83.3
2

Result 13.04 26.07 10.43 20.88

To calculate the value of Chi Square, add up the results for each cell--Total=70.42

DEGREES OF FREEDOM We cannot interpret the value of the Chi Square statistics by itself. Instead, we must put it into a context. In theory, the value of the Chi Square statistic is normally distributed; that is, the value of the Chi Square statistics looks like a normal (bell-shaped) curve. Thus we can use the properties of the normal curve to interpret the value obtained from our calculation of the Chi Square statistic. If the value we obtain for Chi Square is large enough, then we can say that it indicates the level of statistical significance at which the relationship between the two variables can be presumed to exist. However, whether the value is large enough depends on two things: the size of the contingency table from which the Chi Square statistic has been computed; and the level of alpha that we have selected. The larger the size of the contingency table, the larger the value of Chi Square will need to be in order to reach statistical significance, if other things are equal. Similarly, the more stringent the level of alpha, the larger the value of Chi Square will need to be, in order to reach statistical significance, if other things are equal. The term "degrees of freedom" is used to refer to the size of the contingency table on which the value of the Chi Square statistic has been computed. The degrees of freedom is calculated as the product of (the number of rows in the table minus 1) times (the number of columns in the table minus ).

For a table with two rows of cells and two columns of cells, the formula is: df = (2 - 1) x (2 - 1) = (1) x (1) = 1

For a table with two rows of cells and three columns of cells, the formula is: df = (3 - 1) x (2 - 1) = (2) x (1) = 2 For a table with three rows of cells and three columns of cells, the formula is: df = (3 - 1) x (3 - 1) = (2) x (2) = 4 The level of alpha can vary, but the smaller the value, the more stringent the requirement for reaching statistical significance becomes. Alpha levels are often written as the "p-value", or "p=.05." Usual levels are p=.05 (or the chance of one in 20 of making an error), or p=.01 (or the chance of one in 100 of making an error), or p=.001 (or the chance of one in 1,000 of making an error). When reporting the level of alpha, it is usually reported as being "less than" some level, using the "less than" sign or <. Thus, it is reported as p<.05, or p<.01; unless you are reporting the exact p-value, such as p=.04 or p=.22.

DISTRIBUTION TABLES Once we have the calculated value of the Chi Square statistic, and the degrees of freedom for the contingency table, and the desired level for alpha, we can look up the normal distribution for Chi Square in a table. There are many tables available in statistics texts for this purpose. In the table, find the degrees of freedom (usually listed in a column down the side of the page). Next find the desired level of alpha (usually listed in a row across the top of the page). Find the intersection of the degrees of freedom and the level of alpha, and that is the value which the computed Chi Square must equal or exceed to reach statistical significance. For example, for df=2 and p=.05, Chi Square must equal or exceed 5.99 to indicate that the relationship between the two variables is probably not due to chance. For df=4 and p=.05, Chi Square must equal or exceed 9.49.

INTERPRET THE RESULTS If the computed value for Chi Square equals or exceeds the value indicated in the table for the given level of alpha and degrees of freedom, then the researcher can assume that the observed relationship between the two variables exists (at the specified level of probability of error, or alpha), and reject the null hypothesis. This gives support to the research hypothesis. The computed value of Chi Square, at a given level of alpha and with a given degree of freedom, is a type of "pass-fail" measurement. It is not like a measure of association, which can vary from 0.0 to (plus or minus) 1.0, and which can be interpreted at every point along the distribution. Either the computed value of Chi Square reaches the required level for statistical significance or it does not.

It is important to note that Chi Square, like other tests for statistical significance:

1) does not indicate the strength of an association between two variables 2) does not indicate the direction of an association between two variables 3) does not indicate the probability of a Type I error 4) does not take into account the reliability and validity of the research 5) does not provide absolute, conclusive proof of a relationship To recap, for the example above: 1) state the research hypothesis: There is a relationship between the type of training program attended and the job placement success of trainees 2) state the null hypothesis: There is no relationship between the type of training program attended and the job placement success of trainees 3) calculate the test for statistical significance Chi Square=70.42 4) calculate the degrees of freedom of the contingency table df=1 5) select the level of alpha p=.05 6) look up the Chi Square value in the table at p=.05 and df=1 Chi Square=3.84 7) interpret the result The computed value of Chi Square (70.42) exceeds the value in the table for p=.05 and df=1 (Chi Square=3.84). Therefore, we can reject the null hypothesis (with a 5% probability of error) and accept the research hypothesis that a relationship exists between type of training program attended and the job placement success of trainees. Using T-Tests T-Tests are tests for statistical significance that are used with interval and ratio level data. Ttests can be used in several different types of statistical tests: 1) to test whether there are differences between two groups on the same variable, based on the mean (average) value of that variable for each group; for example, do students at private schools score higher on the SAT test than students at public schools? 2) to test whether a group's mean (average) value is greater or less than some standard; for example, is the average speed of cars on freeways in California higher than 65 mph? 3) to test whether the same group has different mean (average) scores on different variables; for example, are the same clerks more productive on IBM or Macintosh computers? To calculate a value of t, a) state the research hypothesis; b) state the null hypothesis; c) stipulate whether the t-test will be a one-tailed test or a two-tailed test for significance

d) e) calculate t

select

the

level

of

alpha

To calculate a value of t, a) state the research hypothesis; The average salary of male graduate assistants is higher than the average salary of female graduate assistants at CSULB. b) state the null hypothesis; There is no difference in the average salary of male and female graduate assistants at CSULB. c) select the level of alpha select a value for alpha, such as p=.05, p=.01, or p=.001 d) stipulate whether the t-test will be a one-tailed test or a two-tailed test for significance Like other statistics, the t-test has a distribution that approaches the normal distribution, especially if the sample size is greater than 30. Since we know the properties of the normal curve, we can it to tell us how far away from the mean of the distribution our calculated t-score is. The normal curve is distributed about a mean of zero, with a standard deviation of one. A tscore can fall along the normal curve either above or below the mean; that is, either plus or minus some standard deviation units from the mean. A t-score must fall far from the mean in order to achieve statistical significance. That is, it must be quite different from the value of the mean of the distribution, something that has only a low probability of occurring by chance if there is no relationship between the two variables. If we have chosen a value of p=.05 for alpha, we look for a value of t that falls into the extreme 5% of the distribution. If we have a hypothesis that states the expected direction of the results, e.g., that male graduate assistant salaries are higher than female graduate assistant salaries, then we expect the calculated t-score to fall into only one end of the normal distribution. We expect the calculated t-score to fall into the extreme 5% of the distribution. If we have a hypothesis, however, that only states that there is some difference between two groups, but does not state which group is expected to have the higher score, then the calculated tscore can fall into either end of the normal distribution. For example, our hypothesis could be that we expect to find a difference between the average salaries of male and female graduate assistant members (but we do not know which is going to be higher, or which is going to be lower). For a hypothesis which states no direction, we need to use a "two-tailed" t-test. That is, we must look for a value of t that falls into either one of the extreme ends ("tails") of the distribution. But since t can fall into either tail, if we select p=.05 for alpha, we must divide the

5% into two parts of 2-1/2% each. So a two-tailed test requires t to take on a more extreme value to reach statistical significance than a one-tailed test of t. e) calculate t A t-score is calculated by comparing the average value on some variable obtained for two groups; the calculation also involves the variance of each group and the number of observations in each group. For example, Table 3. Male and Female Graduate Assistant Salaries at CSULB

MaleGraduate Assistants Number of 403 observations Mean $17,095 Standard 6329 Deviation Variance 40045241

Female Graduate Assistants 132 $14,885 4676 21864976

To calculate t, 1) subtract the mean of the second group from the mean of the first group 2) calculate, for each group, the variance divided by the number of observations minus 1 3) add the results obtained for each group in step two together 4) take the square root of the results of step three 5) divide the results of step one by the results of step four For example, 1) subtract the mean of the second group from the mean of the first group 17095-14885=2210 2) calculate, for each group, the variance divided by the number of observations minus 1 Male graduate assistants: [40056241 / (403-1)] = [40056241 / (402)] = 99642 Female graduate assistants: [21864976 / (132-1)] = [21864976 / (131)] = 166908 3) add the results obtained for each group in step two together 99642+166908=266550 4) take the square root of the results of step three square root of 266550=516.28 5) divide the results of step one by the results of step four 2210/516.28=4.28 To interpret the results, f) calculate the degrees of freedom

g) look up h) interpret the value of t Degrees of freedom

the

value

in

the

table

The degrees of freedom for the t-test is calculated by adding up the number of observations for each group, and then subtracting the number two (because there are two groups). For example, (403 + 132 - 2) = 533 Distribution of T The values of t are printed in tables in most statistics texts. The values of the degrees of freedom are listed in a column down the side, and the values of alpha (p-value) are listed in a row across the top. There are different tables for one-tailed and two-tailed tests of t. Find the correct table for the number of tails. Then find the intersection of the degrees of freedom and the value of alpha in the table. That value is the value that the calculated t-score must equal or exceed to indicate statistical significance. For a one-tailed test of t, with df=533 and p=.05, t must equal or exceed 1.645. For a two-tailed test of t, with df=533 and p=.05, t must equal or exceed 1.960. Interpret the value of t If the computed t-score equals or exceeds the value of t indicated in the table, then the researcher can conclude that there is a statistically significant probability that the relationship between the two variables exists and is not due to chance, and reject the null hypothesis. This lends support to the research hypothesis. In this example, the computed t-score of 4.28 exceeds the table value of t, so we can reject the null hypothesis of no relationship between graduate assistant gender and graduate assistant pay, and instead accept the research hypothesis and conclude that there is a relationship between graduate assistant gender and graduate assistant pay. Remember, however, that this is only one statistic, based on just one sample, at one point in time, from one research project. It is not absolute, conclusive proof that a relationship exists, but rather support for the research hypothesis. It is only one piece of evidence, that must be considered along with many other pieces of evidence on the same subject.

REPORTING TESTS OF STATISTICAL SIGNIFICANCE In research reports, tests of statistical significance are reported in three ways. First, the results of the test may be reported in the textual discussion of the results. Include: 1) the hypothesis 2) the test statistic used and its value 3) the degrees of freedom

4) the value for alpha (p-value) For example, Workers in organizations with non-authoritarian management styles were found to have higher levels of job satisfaction than workers in organizations with authoritarian management styles (Chi Square=50.57, df=4, p<.05). The average salary of male graduate assistants is higher than that for female graduate assistants (t=4.28, df=533, p<.05). No differences were found in job placement rates between vocational education programs and work skills programs (Chi Square=1.2, df=1, p>.05). A second method of reporting the results of tests for statistical significance is to report the test and its value, the degrees of freedom, and the p-value at the bottom of the contingency table or printout showing the data on which the calculations were based. Table 1. Job Placement by Type of Training (Observed Frequencies)

Placed in a Job?

Yes No Total

Type of Training Vocational Education 175 25 200

Work Training 125 125 250

Skills Total 300 150 450

Chi Square=70.42, df=1, p<.05 Table 3. Male and Female Graduate Assistant Salaries at CSULB

Male Graduate Assistants Number of 403 observations Mean $17,095 Standard 6329 Deviation Variance 40045241 t=4.28, df=533, p<.05

Female Graduate Assistants 132 $14,885 4676 21864976

The third way to report tests of statistical significance is to include them in tables showing the results of an extended analysis of the data, including a number of variables. For example, here are some results from a study of older Hispanic women in El Paso, TX, and Long Beach, CA.

Table

4.

Characteristics

of

Workshop

Participants

Age

40

and

Older

Characteristics

El Paso Long Beach value (N=83) (N=131) t Mean age 60.5 years 68.7 years 2.1* Ethnic self-identification 97.2 89.7 0.9 Mexican-American Language preference 68.5 52.3 3.2** Spanish only *t **t significant significant at at

of

p<.05 p<.01

Final Comments Tests for statistical significance are used to estimate the probability that a relationship observed in the data occurred only by chance; the probability that the variables are really unrelated in the population. They can be used to filter out unpromising hypotheses. Tests for statistical significance are used because they constitute a common yardstick that can be understood by a great many people, and they communicate essential information about a research project that can be compared to the findings of other projects. However, they do not assure that the research has been carefully designed and executed. In fact, tests for statistical significance may be misleading, because they are precise numbers. But they have no relationship to the practical significance of the findings of the research. Finally, one must always use measures of association along with tests for statistical significance. The latter estimate the probability that the relationship exists; while the former estimate the strength (and sometimes the direction) of the relationship. Each has its use, and they are best when used together.