Beruflich Dokumente
Kultur Dokumente
for
Prepared by:
11/12/2015
Page 1
I Raise the Rates Program Evaluation Plan
TABLE OF CONTENTS
1. INTRODUCTION 3
EVALUATION PURPOSE 3
PARTNERS 4
NEED 4
CONTEXT 5
TARGET POPULATION 5
STAGE OF DEVELOPMENT 5
INPUTS 5
ACTIVITIES 6
CHAMPION RECRUITMENT AND TRAINING 6
PARTNERSHIPS 6
OUTPUTS 6
OUTCOMES 6
LOGIC MODEL 7
3. EVALUATION DESIGN 7
EVALUATION QUESTIONS 7
PARTNER NEEDS 8
4. DATA COLLECTION 8
DATA ENTRY 8
DATA ELEMENTS 9
DATA PROTECTION 9
DATA MANAGEMENT/ACCESS 9
DATA COLLECTION METHOD – EVALUATION QUESTION LINK 11
ANALYSIS 14
DATA INTERPRETATION 14
EVALUATION USE 14
EVALUATION COMMUNICATION 14
REFERENCES 15
Page 2
I Raise the Rates Program Evaluation Plan
1. INTRODUCTION
Page 3
I Raise the Rates Program Evaluation Plan
Evaluation Purpose
Partners
Partners for this evaluation include:
The American College of Physicians (ACP) is a national
organization of internists — physician specialists who apply scientific
knowledge and clinical expertise to the diagnosis, treatment, and
compassionate care of adults across the spectrum from health to
complex illness. Through its Center for Quality, ACP develops quality
improvement (QI) programs for physicians and their health care
teams, including the refashioning of the registered practice-based
research network—ACP Quality Connect—into a QI network. This QI
network is built on individual initiatives focused in various clinical
areas, including the I Raise the Rates initiative which seeks to
increase national adult immunization (AI) rates. ACP serves as
consultants in developing this evaluation plan.
The Louisiana State University Health Care Services Division (LSU HCSD), a
division of LSU Health, is jointly based on the Baton Rouge and New Orleans LSU
campuses. LSU HCSD is an academic healthcare organization that operates Lallie Kemp
Medical Center, an integrated public safety-net hospital and clinic system in
Independence LA, and supports accountable care services in collaboration with its public-
private partners that operate the other LSU hospitals. LSU HCSD’s Center for Healthcare
Effectiveness studies and leverages the use of clinical data for population health, disease
management, and the application of the next generation informatics and analytics, with
the goal of delivering better, smarter healthcare for the citizens of Louisiana.
School of Public Health (SPH) is one of 6 professional schools
within the Louisiana State University Health Sciences Center – New
Orleans dedicated to teaching, research, and healthcare service
functions throughout Louisiana. SPH assembles teams of
Biostatisticians, Epidemiologists, Environmental/Occupational Health
Scientists, Health Policy/Systems Management scientists, and
Behavioral/Community Health scientists to evaluate multi-level public
Page 4
I Raise the Rates Program Evaluation Plan
When implementing this evaluation, SPH will engage all consultants when
collecting data, interpreting findings, and publishing results.
Need
National immunization efforts have greatly improved life expectancy and
decreased disease morbidity and mortality among American infants,
children and adults. Adult vaccination is highly recommended by major US
health organizations (1), but vaccination rates remain low and fall short of
Healthy People 2020 objectives. (2) Assessing, recommending, and offering
or referring for appropriate vaccinations are key components of preventive
medicine.
Context
Physicians and nurses participate on a voluntarily basis to
improve the adult immunization rates in Louisiana and Arkansas,
Healthcare providers met in May 2015 at the I Raise the Rates
Championship Training meeting with the goal to improve the adult
immunization in their patient populations. Different interventions
programs are available to these providers to educate their patients to
get vaccinated. Immunization rates will be captured over time
through the Med Concert platform, data warehouses or chart reviews.
There are numerous data limitations to be considered and to be
addressed in the final report:
o Self-selection bias of providers being interest in this project to
become immunization champions.
o Different data sources used to capture immunization rates; from
random chart review in small clinics to electronically data entry in
the Med Concert Platform, the warehouse databases and the
Arkansas Immunization Registries
o Immunization champions will come from non-profit and private
hospitals clinics, small and big practices from urban and rural
areas, Federally Qualified Health Clinics (FQHCs) and therefore
should cover patient populations with different racial, ethnic and
social backgrounds. See Attachment B for a map of ACP I Raise the
Rates Champions in Louisiana and Arkansas.
Page 5
I Raise the Rates Program Evaluation Plan
Target Population
Provider study includes all providers participating in immunization
campaign initiatives at hospitals,
Federally Qualified Health Centers, and private practice clinics in Louisiana
and Arkansas. Patient study includes all patients age 19 and older with a
primary care clinic visit at a participating hospital, federally qualified health
center, or private practice clinic in Louisiana and Arkansas.
Stage of Development
Currently, this evaluation plan is in the planning stage. The I Raise the
Rates Adult Immunization Initiative has been in place since January 2015.
Provider Practice assessments and training occurred between March and
May, 2015, for Louisiana and Arkansas champions. The patient and provider
study intervention period started July 1, 2015, and will end June 30, 2016. A
follow-up provider practice assessment will occur six months post
intervention period. Therefore, the total duration for patients is one year,
and for providers is 1 year and 6 months.
1. Enrollment Phase:
a. Providers – January 1, 2015 to September 30, 2015.
b. Patients – None (2014 Benchmark Data).
2. Active Phase:
a. Providers – July 1, 2015 to June 30, 2016.
b. Patients – July 1, 2015 to June 30, 2016.
3. Follow-up Phase:
a. Providers – July 1, 2016 to December 30, 2016.
b. Patients – None.
Inputs
ACP leadership and staff
LSUH-NO HCSD leadership, staff
LSUH-NO SPH Scientists
National Advisory Board
Champions (Hospitals, FQHCs, Private Practice Clinics)
CECity
Other Partners (Pfizer)
Pharmacies
Universities and Colleges
Financial support
Activities
Project oversight and management
Evaluation Plan
o Practice Assessment evaluation
Page 6
I Raise the Rates Program Evaluation Plan
Partnerships
Strengthen relationships with existing partners and build new
partnerships
o Develop Memorandums of Understanding (MOUs) and User
Licensing Agreements (ULAs) among committed providers
Create a community of learning among service providers
Practice support with ACP
Outputs
Number of provider training sessions conducted
Number of providers trained
Number of provider strategy/action plans developed
Number of best practice interventions implemented
Number of provider perception/satisfaction surveys collected
Number of provider practice assessments collected
Summery document of survey findings
Final evaluation report
Outcomes
Short-Term Outcomes
Increase number of participating providers
Increase participating provider use of immunization data
platform
Increase participating provider use of best practice
interventions
Increase adult immunization rates of patient population of
participating practices
Intermediate Outcomes
Evaluation findings shared with partners
o Practice Assessment results
Page 7
I Raise the Rates Program Evaluation Plan
Long-Term Outcomes
Increase patient adult immunization rates of participating
practices
Sustain provider practice commitments and intervention
utilization
Evaluation findings shared with National Advisory Board and at
regional/local conferences
Logic Model
A logic model (Attachment C) was created by planning team members to
illustrate what is being evaluated. The logic model includes: inputs;
activities; outputs; and, short, intermediate, and long term outcomes.
3. EVALUATION DESIGN
Evaluation Questions
Evaluation Evaluation Questions
Focus
Partnership Are the partnership functions conducive to achieving the program
goals and objectives?
Are national and local partners satisfied with the work of this
initiative?
What factors are affecting (positively or negatively) partnership
maintenance?
Program Plan Are program activities being conducted in the manner which they
were intended?
If not, how do they differ and why?
Are there certain activities that need to be added, removed, or
modified?
Are the goals, objectives, and strategies of the program being
implementing as intended? Why or why not?
Program Is there evidence of increase adult immunization rates among
Intervention participating providers
What are the barriers and facilitators to conducting successful
Page 8
I Raise the Rates Program Evaluation Plan
Partner Needs
Developmental evaluation question(s):
Were program activities developed as planned?
Who was responsible for developing the intervention?
Data Source: Meetings, conference calls
4. DATA COLLECTION
Data Entry
Providers:
Survey data will be entered via Survey Monkey by all participating
providers. The provider survey will be launched pre-intervention via
email (Attachment B) with an online link to the survey embedded in
the email. The pre-intervention survey will remain active until a
minimum 50% response rate has been reached (but, no longer than 3
months). A post-intervention provider survey will be launched via
email (Attachment C) with an online link to the survey embedded in
the email. The post-intervention survey will remain active until a
Page 9
I Raise the Rates Program Evaluation Plan
minimum of 50% response rate has been reached (but no longer than
6 months).
Patients:
Participating providers with electronic health records will send
patient data electronically via a monthly data transfer to the data
platform.
Participating providers without an electronic health record will upload
an electronic data file or manually enter data into the data platform.
o Chart abstraction will be conducted for a minimum of 25 patient
medical records by each participating provider before and after
the intervention period.
Data Elements:
Providers:
Demograph Gender Age Race Ethnicity Facility
ics
Clinic E-mail
Patients:
Demograph Medical Zip Code Birthdate Gender Race
ics Record
Number
Ethnicity Facility Clinic Educatio Insuranc
n e
Data Protection
Provider and patient data will be protected by:
1. Implementing a secure electronic file transfer protocol for provider
survey and patient electronic health record data from providers at
participating facilities, and a secure web based interface for manually
entered health record data from providers at participating private
practice clinics, to the ACP data platform.
Page 10
I Raise the Rates Program Evaluation Plan
2. Securely storing all provider and patient data on the ACP data
platform, which is accessible only by authorized users with
confidential user names and passwords through password protected
computers.
3. Implementing a secure electronic file transfer of provider survey data
and patient electronic health record data from the ACP data platform
to the LSU Health New Orleans School of Public Health secure
network.
4. Securely storing patient data on the LSU Health New Orleans School
of Public Health’s secure network, which is accessible only by
authorized users with confidential user names and passwords through
password protected computers.
5. Locking any project related documents in a locked file cabinet in a
locked office at ACP and LSU Health New Orleans School of Public
Health in accordance with federal, state, and institutional policy and
procedure for record maintenance. In compliance of all said
regulations, the documents will be retained for no less than ten years.
Data Management/Access
Original provider survey data reside on a server at a Survey Monkey data
center in the US and Luxembourg (exact physical address unknown). Survey
Monkey can only be accessed by authorized ACP employees.
All study records stored on ACPs server reside at 25 Massachusetts Ave NW,
Suite 700, Washington, DC 20001. ACP servers can only be accessed and
reviewed by authorized ACP employees for submission to SPH.
All study records stored on LSU Health - New Orleans SPH’s server reside
at 2020 Gravier St., 2nd Floor, New Orleans, LA 70112. SPH servers can
only be accessed and reviewed by study personnel.
Page 11
I Raise the Rates Program Evaluation Plan
Focus Evaluation Questions Indicators Source of Data Data Collection Methods Data Analysis
Program Is there evidence of Percent of adults aged 19 years and older Electronic feed transfer Extract data to
Intervention increase adult seen during a primary care visit who were MedConcert for participating providers describe the
immunization rates vaccinated or reported previous receipt of platform at hospitals and FQHCs; or trends and to
among participating an influenza vaccination during the 2014 chart audit manual entry report program
providers? influenza season for participating providers progress.
Percent of adults aged 65 years and older at private practice clinics Changes in the
seen during a primary care visit in 2014 who indicators will
had ever received a pneumococcal be analyzed
vaccination or reported previous receipt of using t-test or
pneumococcal vaccination repeated
Percent of high-risk persons aged 19 to 64 measures
years seen during a primary care visit in analysis of
2014 who had ever received a variance
pneumococcal vaccination or reported procedures, as
previous receipt of pneumococcal appropriate to
vaccination the level of
Percent of adults aged 60 years and older measure.
seen during a primary care visit in 2014 who
had ever received zoster (shingles) vaccine
or reported previous receipt of zoster
(shingles) vaccination
Percent of adults aged 19 years and older
seen during a primary care visit in 2014 who
had ever received a Tdap vaccination or
reported previous receipt of Tdap
vaccination
Page 12
I Raise the Rates Program Evaluation Plan
What are the barriers Number of champions trained Post-workshop Pre and post survey data Analyze survey
and facilitators to Pre-training reflection form: text survey, Pre- will be entered in SAS for data to report
conducting successful comments Training analysis. Text comments average scores,
champion training Post- training survey: satisfaction scores Reflection Form will be documented in frequency, and
program? and text comments Excel for content analysis. other statistics.
Content analysis
will be used for
open-ended
question.
Number of providers attended training Practice Post survey data will be Analyze survey
What are the barriers session Assessment Tool , entered in SAS for data to report
and facilitators to Number of training sessions Post intervention analysis. Text comments average scores,
conducting successful Pre-intervention provider survey provider survey will be documented in frequency, and
training sessions? ( Practice Assessment Tool) scores Excel for content analysis. other statistics.
Post intervention provider survey : Content analysis
satisfaction scores and text comments will be used for
open-ended
question.
What are the barriers Number of participating hospitals and MedConcert Survey data will be Analyze survey
and facilitators to FQHC platform, program entered in SAS for data to report
promote the use of Number of private practice providers records, and analysis. Text comments average scores,
immunization data registered for MedConcert platform Post intervention will be documented in frequency, and
platform by practices Post intervention provider survey : provider survey Excel for content analysis. other statistics.
and clinicians? satisfaction scores and text comments Content analysis
will be used for
open-ended
question.
Page 13
I Raise the Rates Program Evaluation Plan
Focus Evaluation Questions Indicators Data Collection Sources Data Collection Methods Data Analysis
Program Plan Are program activities Extent to which plan activities Program records Program evaluators will Program evaluators
being conducted in are implemented as intended including meeting notes, abstract data from program will measure and
the manner which reports, and other records and other sources report the findings
they were intended? sources. using tables or charts.
Are the goals, Extent to which plan Program records Program evaluators will Program evaluators
objectives, and objectives are as intended including meeting notes abstract data from program will measure and
strategies of the and data collected from records and/or datasets to report the findings
program being champion training compare with original plan, using tables or charts.
implementing as surveys, pre and post and other evidence-based
intended provider surveys practices.
Partnerships Are the partnership Extent to which program Program records Program evaluators will Program evaluators
functions conducive promote collaboration and including meeting notes abstract data from program organize information
to achieving the learning: coaching , technical MOU, and other sources records and other sources and report the
program goals and assistance, and live/virtual descriptive statistics.
objectives? communication and
exchange
Representativeness of key
partners ( ACP chapters,
regional health care systems,
and other health care
leaders)
Are national and local Partners’ satisfaction and Reports/minutes, pre Program evaluators will Program evaluators list
partners satisfied with feedback and post provider abstract data from program all responses, report
the work of this surveys records and surveys. the descriptive
initiative? statistics and
summarizes findings
What factors are Number of existing and new MedConcert platform, Program evaluators will Program evaluators
affecting (positively or champions pre and post provider abstract data from program organize information
negatively) Number of new MedConcert surveys records and surveys. and report the
partnership platform users descriptive statistics.
maintenance?
Page 14
I Raise the Rates Program Evaluation Plan
Data Analysis
The investigators involved in this project plan to use SAS Version 9.4 and
Minitab 17 to summarize the data using descriptive statistics followed by
multivariate analyses, and statistical process control (SPC) techniques for
quality improvement. Process and outcome evaluation data will include
documentation of barriers and positive organizational outcomes as provided
by project staff. Changes in the primary and secondary outcomes will be
analyzed using Chi-square, logistic regression, or repeated measures
analysis of variance procedures, as appropriate to the level of measure.
Interpretation
Upon initial analysis by SPH, all partners will be involved in drawing,
interpreting, and justifying conclusions from preliminary data via
conference meetings.
Evaluation Use
Data from all participating immunization providers will be analyzed to see if
adult immunization rates increased in their practices. In order to identify
target populations for future immunization projects data will be stratified by
state, provider and patient specific variables and reasons for not having
received recommended vaccines. The data analysis will identify subgroups
where vaccination rates did not improve.
Evaluation Communication
Preliminary results will be shared and discussed with ACP before producing
a final report. The final report and power point presentations through
webinars will be used to share the findings and conclusions with provider
champions. In order to reach a wider healthcare provider audience, findings
will be published in 1-2 selected medical journals.
Page 15
I Raise the Rates Program Evaluation Plan
REFERENCES
Page 16
I Raise the Rates Program Evaluation Plan
NOTES
Page 17