Power and Sample Size Calculation

By Gayla Olbricht and Yong Wang

Definition and Application Statistical power is defined as the probability of rejecting the null hypothesis while the alternative hypothesis is true. Factors that affect statistical power include the sample size, the specification of the parameter(s) in the null and alternative hypothesis, i.e. how far they are from each other, the precision or uncertainty the researcher allows for the study (generally the confidence or significance level) and the distribution of the parameter to be estimated. For example, if a researcher nows that the statistics in the study follow a ! or standard normal distribution, there are two parameters that he"she needs to estimate, the population mean (#) and the population variance ($%). &ost of the time, the researcher now one of the parameters and need to estimate the other. 'f that is not the case, some other distribution may be used, for example, if the researcher does not now the population variance, he"she can estimate it using the sample variance and that ends up with using a ( distribution. 'n research, statistical power is generally calculated for two purposes. ). 't can be calculated before data collection based on information from previous research to decide the sample size needed for the study. %. 't can also be calculated after data analysis. 't usually happens when the result turns out to be non*significant. 'n this case, statistical power is calculated to verify whether the non*significant result is due to really no relation in the sample or due to a lac of statistical power.

Statistical power is positively correlated with the sample size, which means that given the level of the other factors, a larger sample size gives greater power. +owever, researchers are also faced with the decision to ma e a difference between statistical difference and scientific difference. ,lthough a larger sample size enables researchers to find smaller difference statistically significant, that difference may not be large enough be scientifically meaningful. (herefore, as consultants, we would li e to recommend that our clients have an idea of what they would expect to be a scientifically meaningful difference before doing a power analysis to determine the actual sample size needed.

Calculation of Stati tical Power (he power is a probability and it is defined to be the probability of rejecting the null hypothesis when the alternative hypothesis is true. ,fter plugging in the re-uired information, a researcher can get a function that describes the relationship between statistical power and sample size and the researcher can decide which power level they prefer with the associated sample size. (he choice of sample size may also be constrained by factors such as the financial budget the researcher is faced with. .ut generally consultants would li e to recommend that the minimum power level is set to be /.0/. 'n some occasions, calculation of power is simple and can be done by hand. Statistical software pac ages such as S,S also offers a way of calculating power and sample size. (he researchers must have some information before they can do the power and sample size calculation. (he information includes previous nowledge about the

parameters (their means and variances) and what confidence or significance level is needed in the study.

!and Calculation" 1e will use an example to illustrate how a researcher can calculate the sample size needed for a study. 2iven that a researcher has the null hypothesis that #3#/ and alternative hypothesis that #3#)4 #/, and that the population variance is nown as $%. ,lso, he nows that he wants to reject the null hypothesis at a significance level of 5 which gives a corresponding ! score, called it !5"%. (herefore, the power function will be 67!8 !5"% or !9 *!5"%:#);3)*<=!5"%*(#)*#/)"($"n)>?<[-!5"%*(#)*#/)"($"n)>. (hat is a function of the power and sample size given other information nown and the researcher can get the corresponding sample size for each power level. For example, if the researcher learns from literature that the population follows a normal distribution with mean of )// and variance of )// under the null hypothesis and he"she expects the mean to be greater than )/@ or less than A@ under the null hypothesis and he"she wants the test to be significant at A@B level, the resulting power function would beC 6ower3)*<=).AD*()/@*)//)"()/"n)>?<[-).AD*(A@*)//)"()/"n)>, which is, 6ower3)*<=).AD*n"%>?<[-).AD?n"%>. (hat function shows a relationship between power and sample size. For each level of sample size, there is a corresponding sample size. For example, if n3%/, the corresponding power level would be about /.AE, or, if the power level is /.A@, the corresponding sample size would be )D.

# ing Stati tical Pac$age %SAS& Statistical pac ages li e S,S enables a researcher to do the power calculation easily. (he procedure in which power and sample size are calculated is specified in the following text. 'n S,S, statistical power and sample size calculation can be done either through program editor or by clic ing the menu the menu. 'n the latter, a set of code is automatically generated every time a calculation is done. P'OC POW(' and G)*POW(' 6GHI 6H1JG and 2K&6H1JG are new additions to S,S as of version A./. ,s of this writing, S,S A./ is not currently installed on '(a6 machines, but it can be installed on your home computer using dis s available in Steward .)L. &a e sure to bring your 6urdue 'M. (he table on the following page (ta en from the S,S help file) shows the types of analyses offered by 6GHI 6H1JG. ,t least one statement is re-uired. (he syntax within each statement varies, however, there is some syntax common to all. (hese common features will be expressed by an example using a paired t*test. &ore information on each procedure can be found in the S,S help file. 'n the example, assume that a pilot study has been done, and that the standard deviation of the difference between the two groups has been found to be @, with a mean difference of %. 1eNd li e to calculate the re-uired sample size for an experiment with 0/B power.
proc power; pairedmeans meandiff stddev = npairs = test=diff = 2 5 .

run;
Analysis

power = .80;

Statement &OK(GJ2 HQJIHGG HQJIHGG HQJS,&6KJFGJR HQJS,&6KJFGJR HQJS,&6KJFGJR HQJS,&6KJ&J,QS HQJS,&6KJ&J,QS HQJS,&6KJ&J,QS HQJS,&6KJ&J,QS

Options

&ultiple linear regressionC (ype ''' F test IorrelationC FisherPs z test IorrelationC t test .inomial proportionC Jxact test .inomial proportionC z test .inomial proportionC z test with continuity adjustment Hne*sample t test Hne*sample t test with lognormal data Hne*sample e-uivalence test for mean of normal data Hne*sample e-uivalence test for mean of lognormal data

M'S(3F'S+JG! M'S(3( (JS(3JS,I( (JS(3! (JS(3,MT! (JS(3( (JS(3( M'S(3KH2QHG&,K (JS(3JRO'U (JS(3JRO'U M'S(3KH2QHG&,K I'3( (JS(3IHQ(G,S( (JS(3HUJG,KK

Ionfidence interval for a mean Hne*way ,QHU,C Hne*degree*of*freedom contrast Hne*way ,QHU,C Hverall F test &cQemar exact conditional test &cQemar normal approximation test 6aired t test 6aired t test of mean ratio with lognormal data 6aired additive e-uivalence of mean difference with normal data 6aired multiplicative e-uivalence of mean ratio with lognormal data Ionfidence interval for mean of paired differences 6earson chi*s-uare test for two independent proportions FisherPs exact test for two independent proportions Ki elihood ratio chi*s-uare test for two independent proportions (wo*sample t test assuming e-ual variances (wo*sample Satterthwaite t test assuming une-ual variances (wo*sample pooled t test of mean ratio with lognormal data (wo*sample additive e-uivalence of mean difference with normal data (wo*sample multiplicative e-uivalence of mean ratio with

HQJS,&6KJ&J,QS HQJ1,V,QHU, HQJ1,V,QHU, 6,'GJMFGJR 6,'GJMFGJR 6,'GJM&J,QS 6,'GJM&J,QS 6,'GJM&J,QS 6,'GJM&J,QS

M'S(3QHG&,K (JS(3M'FF (JS(3G,('H (JS(3JRO'UWM'FF (JS(3JRO'UWG,('H

6,'GJM&J,QS (1HS,&6KJFGJR (1HS,&6KJFGJR (1HS,&6KJFGJR (1HS,&6KJ&J,QS (1HS,&6KJ&J,QS (1HS,&6KJ&J,QS (1HS,&6KJ&J,QS

I'3M'FF (JS(36I+' (JS(3F'S+JG (JS(3KGI+' (JS(3M'FF (JS(3M'FFWS,(( (JS(3G,('H (JS(3JRO'UWM'FF

(1HS,&6KJ&J,QS

(JS(3JRO'UWG,('H

lognormal data (wo*sample confidence interval for mean difference Kog*ran test for comparing two survival curves 2ehan ran test for comparing two survival curves (arone*1are ran test for comparing two survival curves (1HS,&6KJ&J,QS (1HS,&6KJSOGU'U,K (1HS,&6KJSOGU'U,K (1HS,&6KJSOGU'U,K I'3M'FF (JS(3KH2G,QX (JS(32J+,Q (JS(3(,GHQJ1,GJ

Power and Sample Size Calculation # ing SAS *enu 6ower and sample size can also be calculated using the menu in S,S. 1hen using the menu, the user should specify the chosen design for the underlying project, and then fill in the re-uired parameters needed to do the calculation for each design. (he general procedure of using the menu is as followsC )). Hpen S,S %). 2o to the enhanced editor window. F). Ilic the P olution P button on the menu. L). 'n the submenu, clic Panaly i P. @). 'n the next submenu, clic Panaly tP, then a new window will pop*up. D). 'n the new window, clic P tati tic P button on the menu. E). Select PSample izeP, then select the design you want to use. (the designs available in that menu includeC one*sample t*test, paired t*test, two sample t*test and one*way ,QHU,). 0). ,fter you select the design another window pops*up and as s you to input the needed options and parameters. 'f you need to now the needed sample size for your research, you can select P+ per GroupP, then input number of treatment , corrected um of -uare, the tandard de.iation and the alpha level. 'f the researcher wants to calculate the sample size corresponding to each D

power level, he"she may want to specify the range and interval of power level in the YPowerN row in the menu. (he corrected sum of s-uares (ISS) is calculated as the sum of the s-uared distance from each treatment mean to the grand*mean. For example, there are two treatments with mean of )/ and %/, respectively. (hat gives us a grand mean of ()/?%/)"%3)@ (assuming e-ual cell size). (herefore, the corrected sum of s-uares isC ()/*)@)%?(%/*)@)%3@/. Hnce the re-uest for calculation is submitted, S,S will pop*up a window which includes a table of power level and corresponding sample size. Vou can also as S,S to generate a curve showing the relation between power level and sample size. ,nother important feature of S,S menu is that you can generate the code by which you use to do the power calculation and it will be displayed in another window. (/ample Output ,n example is shown below using the ISS mentioned above and assuming a one*way ,QHU, design is used. 1e also assume that the standard deviation is %/ and the alpha is /./@. 1e want to find out the corresponding sample size for each power level ranging from /.0 to /.AA at /./) intervals. (he outputs should loo li e the followingC
One-Way ANOVA # Treatments = 2 CSS of Means = 50 Standard Deviation = 20 Al !a = 0"05 N er #o$er %ro& 0"'00 0"'*0 0"'20 () (( ('

0"'+0 (, 0"')0 -* 0"'50 -+ 0"'(0 -5 0"'-0 -' 0"''0 '0 0"',0 '+ 0",00 '( 0",*0 ', 0",20 ,2 0",+0 ,( 0",)0 *00 0",50 *05 0",(0 **2 0",-0 **, 0",'0 *+0 0",,0 *)'

(he output above gives the re-uired sample size per group for each power level. For example, if we want a power level of /.A, we actually need 0DZ%3)E% subjects in the sample. (/ample from Con ulting Ser.ice Client 0 Pro1ect

Ionsider a hypothetical study in which the goal is to determine the effectiveness of a certain drug in lowering diastolic blood pressure. , group of men and women will be randomly assigned to either receive the drug or to receive a placebo. (his design can be analyzed as a one*way ,QHU, with four groupsC ()) men not ta ing the drug, (%) men ta ing the drug, (F) women not ta ing the drug, and (L) women ta ing the drug. , previous study indicates that the means for each of these groups might be AF, EL.D, 0D.E, and ED.@ respectively. (hat study examined a similar -uestion and although the means may not be exact, they are a good estimate. (he standard deviation in diastolic blood pressure between subjects was %E for that study. (he researcher planning the study would li e to now the total number of subjects that will be needed to detect a practical difference in the diastolic blood pressure between subjects receiving the drug and the subjects not receiving the drug. , significance level of /./@ and a power of /.0 are desired. (he following S,S code was used to arrive at an appropriate sample size given these conditions.
proc power; onewayanova test=constrast groupmeans = 93 | 74.6 | 86.7 | 76.5 stddev = 27 alpha = 0.05 contrast = (-1 1 -1 1) ntotal =. power = 0.8; plot =power min=0.! ma =".0; run;

(/planation of code2 onewayanova * Mesignates the type of design. test3contrast * Mesignates the type of test for which the power will be computed. 'n this case, a contrast which will compare subjects receiving the drug to subjects not receiving the drug is the main test of interest. A

groupmeans * Step where each of the four group means are listed. 'f other magnitudes of mean difference were of interest, these could be modified. stddev * Step where the standard deviation is specified. alpha * Step where the significance level is specified. contrast * Specifies the details of the contrast. 'n this case, the contrast will be between groups ) and F (men and women not ta ing the drug) and groups % and L (men and women ta ing the drug). 'f a contrast that compares men and women were of interest, this step could readC contrast3 () ) *) *)). ntotal 3. * Specifies that the total sample size is what needs to be calculated. (his could be given and the power for that particular sample size could be calculated instead. power 3/.0 * Step where the desired power is specified. (his could be calculated (designated with at P.P) if the sample size is given. plot x3power min3/.D max3)./ * (his statement provides a power curve which will display power ranging from /.D to )./ on the x*axis and the sample size which corresponds to that power on the y*axis. A+O3A Power Calculation 'e ult (he 6H1JG 6rocedure Single MF Iontrast in Hne*1ay ,QHU, Fixed Scenario Jlements &ethod Iontrast Ioefficients ,lpha 2roup &eans Standard Meviation Qominal 6ower Qumber of Sides Qull Iontrast Ualue 2roup 1eights Jxact *) ) *) ) /./@ AF EL.D 0D.E ED.@ %E /.0 % / ))))

Iomputed Q (otal ,ctual Q

)/

6ower /.0/E

(otal ))D

From this output, it was determined that ))D subjects total or ))D"L3%A subjects per group will be needed to achieve a power of /.0/E for the specified test.

,fter seeing this result, the researcher may be willing to either recruit more subjects to achieve a higher power or recruit less subjects and sacrifice a small reduction in power. (o visualize these inds of tradeoffs, two power curves were constructed. (he first curve (i), plots sample size as a function of power. (he S,S code for this plot was given previously. (his curve would be useful if the researcher nows a range of power that is desired. From this graph, we can see that lowering the power to /.E@ results in a sample size of around )//, whereas increasing the power to /.0/ results in a sample size of around )F/.

i" Power Cur.e for A+O3A" Sample ize .er u power" ))

,lternatively, if the researcher nows a range of sample sizes that is practical in terms of cost and availability of subjects, a different type of power curve might be more useful. (his curve (ii) graphs power as a function of sample size. From this curve, we can see that if only 0/ subjects complete the study, the power will be reduced to around /.D@. 'f subjects are li ely to withdraw from the study, this curve could also be useful for hypothetical situations involving different numbers of subjects dropping out given a certain number of subjects are recruited in the beginning of the study. (he S,S code for this curve (ii) is the same as for the previous curve (i) except a number must be specified for ntotal, a dot must be specified for power, and the plot statement must change to plot x3n min3%/ max3)%/.

ii" Power Cur.e for A+O3A" Power .er u

ample ize"

)%

4uture Plan for the Pro1ect (he next step of the project is trying to find out how to do power calculation on different inds of designs and how to do power analysis on other software pac ages other than S,S.

)F