Selecting Right Statistics

Hyungjin Myra Kim, Sc.D.

The University of Michigan

Choosing an Analytic Method (1)

First, analytic plan should be considered while
planning the study.
What do you plan to study (or measure)?
Primary outcome measure determines the type
of dependent variable

Continuous (ex: hours of sleep)

Dichotomous (ex: binge drinking or not)
Ordinal (ex: depression diagnosis)
Categorical (ex: choice of treatment)
Time to event (ex: time to relapse)

Choosing an Analytic Method (2)

Sometimes, there is no dependent variable

Factor analysis
Cluster analysis
Higher-way contingency table analyses
Agreement (kappa)
Correlation analysis (correlation coefficient)
Accuracy (sensitivity, specificity)

(We will not discuss the above today.)

Choosing an Analytic Method (3)

Study design
Do you have a primary comparison?
Determines the nature of the primary predictor
variable (independent variable)
(ex) 2 group or 3 group comparison?
(ex) Evaluating the relationship between happiness
to ratio of leisure to work hours
How often do you plan to measure?
X-sectional, longitudinal, x-over
Determines the number of dependent variables
(ex) pre/post has measurements twice per person

The choice of analysis will also depend on

Unvariate vs. bivariate analysis
Bivariate vs. multivariate analysis
Potential confounder?
Adjust for covariates?
Data skewed or sample size small?
Transformation
Parametric vs. non-parametric

Dependent Variable (Outcome)

Study Designs
Pre/Post

Continuous
Effect of nightly exercise on hrs
of sleep before/after in
insomniacs

Matched pairs Mastectomy vs. Lumpectomy on

QOL in patients matched by age
& family history

Binary (yes/no)
Patient satisfaction before vs.
after color change in hospital
ward
Mastectomy vs. Lumpectomy on
survival in patients matched by
age & family history

1-group

Cholesterol in diabetic patients:

Is it higher than general public?

Depression in substance
abusers

2-group

Writing skill between teaching

methods A vs. B

Comparison of drugs A vs. B on

relapse to heavy drinking

2-group,
pre/post

Weight before/after in exercise

vs. no exercise group

Satisfaction before & after

between 2 skin products

3-group

Comparing effectiveness of three

drugs on cholesterol

Pain reduction in three

different pain relief medication

Does pack-year of smoking

predict Cognitive deficit?

Is average nightly sleep

predictive of hair loss?

Continuous
Predictor

What Type of Analysis?

Descriptive
Numerical tables of means, counts,
proportion
Graphical - histograms, box plots, scatter
plots, etc.
Inferential
Estimation Point estimates/Confidence
Intervals
Hypothesis Tests

Analytic Methods
Dependent Variable (Outcome) Type

Study Design

Continuous
(multiple regression for
multivariate analysis)

Binary (yes/no)
(logistic regression for
multivariate analysis)

Pre/Post

Paired t-test

McNemars test

Matched pairs

Paired t-test

McNemars test

1-group

One-group t-test

One proportion test

2-group*

Two-group t-test

Two proportion test or

Chi-square Test

2-group, pre/post*

Analysis of Covariance or
multiple regression

Repeated measures
logistic regression

3-group*

Analysis of Variance

Chi-square test

Continuous
Predictor

Simple regression

Logistic regression

* Bivariate relationships

Binary Dependent Variable

Descriptive Statistics: Proportion
To estimate a proportion or prevalence, subjects must
be a representative sample from the population.
Assuming the subjects are representative and
independent, the rate is estimated as:
p = n/N
where n is the number of subjects with the attribute
and N is the total number of subjects tested (or
studied).

Binary Dependent Variable (2)

When Only One Group is of Interest:
Test
Proportion compared to a null value
one proportion test
Ex) Are substance abusers more likely to be
depressed than general public?
Confidence Interval (95% CI: proportion 1.96*SE)
Ex) Prevalence of depression in substance abusers
Ex) Sensitivity and specificity of a new short
depression instrument compared with the
physicians gold standard depression diagnosis
More on interpretation of 95% CI tomorrow.

Binary Dependent Variable (3)

When Comparing Two Independent Groups:
Ex) Comparing drugs A vs. B on relapse to heavy
drinking
Essentially a 2 by 2 table
Comparative Test
- Chi-square test
- Two proportion test
Comparative Statistics (summary effect size)
- Absolute Difference in Proportions
- Odds Ratios (OR)
- Relative Risks (RR)
For both OR and RR, 1 means no difference
Can calculate 95% CI for any of the above (OR, etc.)

Binary Dependent Variable (4)

When Comparing Two Independent Groups:
If sample size is small
rule of thumb = expected cell count < 5
Comparative Test: Fishers Exact Test
|
A
B|
Total
--------------------------------------------yes |
3
6 |
9
no |
9
2 |
11
-------------------------------------------Total |
12
8 |
20
Pearson chi-square test p-value = 0.028
Fisher's exact test p-value = 0.065

Continuous Dependent Variable

Descriptive Statistics
Mean and Standard Deviation if data are symmetric
Median and Inter-quartile range if data are skewed
Mean can be affected by one very large or one very small
value, and therefore is sensitive to outlying values
Median is robust to an outlying value because it is simply
the value at the center when data are ranked in order.
If mean and median are very different, data are skewed.
Always graphically explore the distribution (e.g., using
histogram, box plot) and choose the appropriate
descriptive statistics
More on mean vs. median tomorrow.

Continuous Dependent Variable (2)

When Only One Group is of Interest:
Test (One mean compared to a null value)
One sample t-test
Ex) Is cholesterol higher in diabetic patients compared
with the general public?
Confidence Interval (95% CI = mean 1.96*SE)
Ex) Sample mean cholesterol = 124
Sample SD = 10, N = 200
95% CI for mean cholesterol = 124 1.96*10/sqrt(200)
= (122.6, 125.4)

Continuous Dependent Variable (3)

When Only One Group is of Interest:
When sample size is small (N<25) or cannot assume that the
dependent variable is interval and normally distributed
Use a Non-parametric Test
One Sample Median Test
- Sign test
- Sign rank test

Continuous Dependent Variable (4)

When Comparing Two Independent Groups:
Test
Two independent group t-test
Ex) Writing skill comparison between teaching methods A vs. B
Comparative Statistics: difference in means
Ex) Difference in mean writing skill scores between
those who were taught with method A vs. method B
Confidence Interval for Difference in Means
95% CI = difference 1.96*SE (of difference)

Continuous Dependent Variable (5)

When Comparing Two Independent Groups:
If sample size is small (N < 25) or cannot assume that the
dependent variable is interval and normally distributed
Use a Non-parametric Test (Test of Median)
Wilcoxon ranksum test (tests equality of medians)

Graphical Methods to Compare Groups: Box Plots

Resting
Heart
Rate

No
Exercise

Mild
Exercise

Strenuous
Exercise

Using Subjects as Their Own Controls:

Cross-Over Designs
Same subject undergoes 2 or more treatments

Advantage

Limitations of reusing the same subject

May not be possible

Carryover effect of treatment need washout
Length of experiment

Order effect

Maximizes power fewest subjects needed

Order should be randomized and balanced

Period effect

Cross-Over Designs (2)

Examples

Pre-post study (poor design, why?)

Ex) Weight before an exercise program and weight
after a month of exercise program

Traditional X-over Study

Ex) Alternating exposure to guided imagery procedure
between stressful situation and a natural relaxing
situation on different days in random order and
assessing the effect on craving

Stressful Image washout period Relaxing Image

Relaxing Image washout period Stressful Image

Ex) Drug A then cross over to B

Cross-Over Designs (2)

Analytic Method
Pre-post study
Analyze change-score or gain-score and treat it as
a one sample problem
Ex) change in weight within a person before and
after the exercise program

Traditional X-over Study

Analysis must first assess carryover effect, order
effect and period effect.
If any effect, then must account for it.

Multiple Comparison: Doing Many Tests

-level (significance level) the probability
of claiming that there is a difference when
there is no true difference
Small is good.
We usually set -level at 0.05.
This means we allow 5% for making the
kind of error where we declare a
significant difference (reject the null
hypothesis) when the result happened by
chance (Type 1 error).

Multiple Comparison (2)

When 2 comparisons, (5%) should be reduced to
adjust for the number of comparisons.
Suppose we are performing two independent
statistical tests, then:
P(of rejecting the 1st when true) is 0.05
P(of rejecting the 2nd when true) is 0.05
What is probability of rejecting at least one?
P(of accepting 1st when true) is 0.95
P(of accepting 2nd when true) is 0.95
Therefore, p(of accepting both)
= 0.95 x 0.95 = 0.9025
That is, p(of rejecting at least one) = 0.0975

Multiple Comparison (3)

Number of
independent tests
1

Probability of rejecting
null hypothesis,
when true
0.05

0.0975

0.143

0.226

10

0.401

If perform enough significant tests, you are sure to

find significant results by chance alone even when
none exists.

Multiple Comparisons: What to do? (4)

For independent tests, one easy way of adjusting the
level of significance is to use:
0.05/k
where k is the number of tests to be performed.
Therefore, instead of 0.05,
When there are 5 tests, use 0.01
When there are 10 tests, use 0.005

Multiple Comparison (5)

When testing a pre-specified relationship, use a
significance level of 5%.
When screening for interesting relationships,
use significance level of 1% so as not to identify
too many false relationships.

Confounding
Example 1: Sex bias in graduate admissions?
(UC, Berkeley, 1973)
Overall:

44% of males admitted

35% of females admitted

Admissions are made by department.

Confounding (2)
Male

Female

Number of
Applicants

Percent
Admitted

Number of
Applicants

Percent
Admitted

A
B

825

62%

108

82%

560

63%

25

68%

C
D

325

37%

593

34%

417

33%

375

35%

E
F

191

28%

393

24%

373

6%

341

7%

Total

2691

45%

1835

30%

Major

Weighted Average:

39%

43%

Confounding (3)
Example2 : Is psychiatric hospitalization rate different
in substance users versus non-users?
Hospitalization
Yes
No
User
20
373 5.1%
Non-user
6
316 1.9%
Substance use looks to be associated with higher
psychiatric hospitalization rate.

User
Non-User

Separated by Bipolar Status

No Bipolar
Bipolar I/II
3
176 1.7%
17 197 7.9%
4
293 1.4%
2
23 8.0%

Confounding (4)
Example 3: Smoking versus MI
Smoker
Non-Smoker
MI
51
54
No MI
43
67
54%
44.6%
OR = 1.47
Male
Female
MI
37
25
14
29
No MI 24
20
19
47
61% 56%
48% 38%
OR = 1.23
OR = 1.19
Smokers have higher MI rate, but the magnitude of the
relative likelihood of MI (measured as odds ratio (OR)) is
larger in the combined data.

Confounding (5)
Example 4:
1) Regression of Happiness on Smoker Group
Coef
SE
p-value
Intercept
65.05
1.48
0.000
Smoke
4.80
2.03
0.020
2) Regression of Happiness on Age
Coef
SE
Intercept
7.48
2.45
Age
1.85
0.07

p-value
0.003
0.000

3) Regression of Happiness on Age and Smoke

Coef
SE
p-value
Intercept
2.65
2.07
0.203
Age
2.08
0.07
0.000
Smoke
-5.25
0.70
0.000

Confounding (6)
Relationship between Happiness and Age

20

100
20

40

40

Happiness
60

60

80

80

100

Without Considering Age, smokers appear to have higher mean

by Smoking Status

20
Not

25

30

Smoke

Y, Smoke == Not

Age

35

40

45

Y, Smoke == Smoke

Increasing age is associated with greater happiness.

Smokers tend to be older, making it look like smoking is associated
with greater happiness when not adjusting for age.
But smokers tend to be less happy than non-smokers given same age.

DevelopingaStatisticalAnalysisPlan
Comparing two groups
Continuous: t-test
Proportion: chi-square test
Comparing multiple groups (continuous): ANOVA
Adjusted for other factors: ANCOVA, or regression
Dichotomous outcome: Logistic regression
Count outcome: Poisson regression
Survival time outcome: Cox regression
Watch for correlated data (repeated measures, clusters
e.g., teeth in the mouth

To Keep in Mind
Typically, multiple appropriate methods are available
to analyze the same data that could yield legitimate
answers.
Try to use at least two different available methods to
confirm your results.
Always look at the raw data and display data
graphically, so learn to choose the right graphical
displays (ex: cross tabs, scatter plots, box plots)
It helps to make sample tables summarizing results
before you start the analysis.