Brian L. Strom, M.D., M.P.H.

Chair and Professor, Department of

Biostatistics and Epidemiology
Director, Center for Clinical Epidemiology
and Biostatistics
George S. Pepper Professor of Public
Health and Preventive Medicine
Professor of Biostatistics and
Epidemiology, Medicine, and Pharmacology
University of Pennsylvania
CCEB
School of Medicine
 Sampling

1. What is an appropriate sample size of
respondents to best determine risk of
sound and look alike proprietary
names in the prescription drug study
group? In a focus group? In a survey
document?
2. Should sample be randomly selected?
Is it important to have a statistical
significance for this type of
CCEB evaluation?
 Sampling

Introduction
Very general principles of
sample size calculations and
sampling
Application of general
principles to this situation
Recommendations for
research, to guide the future
CCEB
 Options in Research
Design
Analytic Studies
Experimental Study
Prospective Cohort Study
Retrospective Cohort
Study
Case-Control Study
Descriptive Studies
Analyses of Secular
Trends
Case Series
Case Reports
CCEB
 Case-Control Studies

Disease
Present Absent
(cases) (controls)
Cohort Studies

Present
(exposed)
A B
Factor

Absent C D
(not exposed)
 Determinants of
Needed
Sample Size
Alpha
Beta
Variability (SD)
Delta--how small a
difference do you want
to be able to detect?
CCEB
 Determinants of Needed
Sample Size: Cohort
Study
Alpha
Beta
Incidence in the unexposed
control group
Delta--how small a RR do you
want to be able to detect?
CCEB
 Determinants of Needed
Sample
Size: Case-Control Study
Alpha
Beta
Prevalence in the undiseased
control group
Delta--how small a RR do you
want to be able to detect?
CCEB
 Samplin
g
Study Sample
Statistical
Inference
Conclusion About a Population
(Association)

Biological Inference

Conclusion About Scientific Theory

CCEB (Causation)
 Application of General
Principles to This
Situation

THE Central Principle

of Research Design: The
question is, what is the
question??

CCEB
 Application of General
Principles to This
Situation
In this situation, there are no
a priori hypotheses being
tested to be able to consider
sample size calculations or
questions of sampling
What is being performed is
essentially qualitative
research
CCEB
 Recommendations for
Research,
to Guide the Future

**Evaluate the current

process in a quantitative
fashion**

CCEB
 General Strategy for
Evaluation
Standardize procedure
Test for reliability/
reproducibility
Test for validity
Make changes in the
procedure accordingly
CCEB
 Standardize
Procedure
Choose among the
current possible
approaches a
standard to be
evaluated more
rigorously
CCEB
Test for Reliability/Reproducibility

Evaluate the same drug names

in the same process with
multiple different groups of
survey prescribers and experts,
to look for whether there is
adequate agreement
If no reliability, validity is
impossible and procedure
should be abandoned
CCEB
 Test for Validity

Gold standard needed:

Drug names rejected in initial
FDA review
Drug names withdrawn due to
problems
Direct measurement of error rate
CCEB
 Retrospective: Drug Names
Rejected in Initial FDA Review

Problem: was the initial

review decision correct?

CCEB
 Retrospective: Drug Names
Withdrawn Due to
Problems
Knowledge of reviewers could
be problematic
Other countries
Years ago, and using new
pharmacists
Was the withdrawal decision a
correct one?
CCEB
 Prospective: Direct
Measurement of Error Rate
Simulate real life situation in a study setting
Choose good and bad options for new
names
Enter possible new names into prescription
entry computer program
Ask large numbers of docs to write orders
Ask large numbers of pharmacists to fill
each rx, entering into prescription entry
computer program
Directly measure resulting error rate
CCEB
 Make Changes in the
Procedure Accordingly
Determine appropriate cutpoint
for expert ratings, ROC curve
vs. gold standard
Determine appropriate sample
sizes through simulation, ie how
many are needed to achieve
results consistent with the gold
standard
Modify processes accordingly
CCEB
 Potential Sources of
Support
FDA extramural funds
AHRQ patient safety funds
NIA pharmacology program
NIGMS pharmacology program

CCEB
 Conclusion
s
Applying a quantitative
approach to evaluating what
has so far been a qualitative
one, could lead to major
changes in the procedure,
and major improvements in
the net results
CCEB
Statitsically incorrect