!

Clinical Decision Support: Seasonal Influenza Immunization in an Ambulatory Care Setting

Margaret Chapman, Raja Ekambaram, and Brad Schwartz
Northwestern University












#

Clinical Decision Support: Seasonal Influenza Immunization in an Ambulatory Care Setting
Introduction
The H1N1 influenza pandemic of 2009 claimed approximately 400,000 lives worldwide (CDC,
2013). As a result, in 2010 the Centers for Disease Control (CDC) Advisory Committee on
Immunization Practices (ACIP) recommended universal yearly influenza vaccination for every
eligible individual older than six months of age. In spite of this, efforts to achieve universal
influenza vaccination have met with limited success. Flu vaccination coverage among children
did not significantly change for the 201112 season compared to the 201011 season, with rates
remaining around 50 percent (CDC, 2013). Statistics are worse for adults, among whom
influenza vaccination coverage decreased by 1.7 percent for the 2011-12 season compared to the
2010-11 season, from 40.5 percent to just 38.8 percent (CDC, 2013). Increased rates of influenza
vaccination would be expected to translate into lower rates of seasonal influenza infections and
decreases in associated morbidity and mortality. In view of this, the CDC has endorsed efforts to
increase influenza vaccination rates through proven methods including strong recommendation
by healthcare providers and reminder strategies such as standing orders, patient reminder and
recall systems, and provider reminders (CDC, 2013).
Electronic medical record (EMR) systems configured with clinical decision support
(CDS) features such as alerts, guidelines, and standing orders can aid providers in ensuring high
quality, evidence-based care (Bates, 2003). The authors hypothesized that a CDS system might
improve rates of seasonal influenza vaccination among children and adults in an outpatient,
family practice setting. The literature on CDS for immunization shows conflicting results.
Immunization reminder EMR alerts have been shown to significantly improve rates of routine
childhood immunization (Fiks, 2007). However, a study by the same investigator concluded that
$

pediatric influenza vaccination rates were not significantly improved in the presence of EMR
alerts (Fiks, 2009). This is in contrast to results in two studies of adult patients, in which rates of
influenza vaccination increased with the use computerized alerts (Dexter, 2001 and Swenson,
2012).
This report describes the development, implementation, and evaluation of a CDSS
designed to increase rates of influenza vaccination in a family practice group. This CDSS was
developed as part of a broader quality improvement initiative that includes patient education,
staff education, and tracking outcomes before and after the intervention. Results will be
submitted as a clinical quality measure (CQM) for Meaningful Use Stage 1 (MU1) and to meet
the CDS requirement for MU2 (CMS, 2012). The primary clinical objective for the CDSS is a
significant increase in seasonal influenza immunization rate compared with previous years.
Stakeholders, Goals, and Objectives
To be effective, CDS should be integrated into the users workflow, should appear at the
appropriate point in the patient encounter, and should offer rapid access to a simple action step
(Fiks, 2006). To achieve this, careful planning and communication are essential. All stakeholders
in the project were identified, along with their roles, motivation to participate, goals for the
project, concerns, and stakes in the outcome (Osheroff, 2012). Stakeholders fell into two main
groups: technical consultants responsible for design and implementation, and clinical/support
staff of the practice who would be the users of the CDSS. One or more representative from each
stakeholder group served on the implementation team. Some team members filled more than one
role; for instance, one provider was also a partner in the practice, and a medical assistant doubled
as a patient/parent, since she and her children are patients of the practice. Clinical team members
were chosen based on their comfort with EMR and technology in general, and their positive view
%

toward implementing the CDSS. Recruiting such champions, who are respected members of
the constituencieswho fully understand and support the CDS efforts, goals and strategies and
can serve as ambassadors (Osheroff, 2012) is critical to the programs success. It is equally
important to identify stakeholders who have negative views toward the project; these may be
individuals who are uncomfortable using technology, who fear the program will create more
work for them, or who resist change. The concerns of resisters were solicited and addressed in
designing and implementing the system. Efforts to involve all stakeholders at all stages in the
development process ensures that CDS is done with them and not to them (Osheroff, 2012).
Identification of stakeholders is also useful for planning schedules and assigning project
responsibilities. An overview of stakeholders in the influenza immunization CDSS appears in
tables one and two.












&

Table 1. Clinical Stakeholders



Stakeholde
r
Role/Participation
in
CDSS
Motivation Goals Stake
in CDS
Concerns
Patients/
Parents
CDSS user/ patient
portal reminders
Vaccine recipients;
consent to vaccinate
minors; reports prior
immunization
Prevent illness
and missed
school/work due
to flu
Preventio
n of
influenza
infection
Low Safety/efficac
y of flu
vaccine
Physician
Partners
Practice owners
Subject matter
experts,
oversight,
champions
Meaningful Use
incentives ($)
Optimize
EMR use
and ROI;
MU for
CDS and
CQM
High Cost, loss of
productivity
Providers
(MD, PNP)
CDSS user EMR
Recommends,
orders flu vaccine;
addresses
questions/concerns;
documents
immunization
refusal
CDC
recommendation
;
pay-for-
performance
incentive
programs;
quality of care
Primary
prevention
of
influenza
for all
eligible
patients

Medium Slowed
workflow,
takes time
from other
patient care,
increased
documentation
Nurse CDSS user EMR
Administers
vaccines;
documents
immunization
Performance
measure; quality
of care
Primary
prevention
of
influenza
for all
eligible
patients
Medium Significant
increased
workload
during busy
flu season
Medical
Assistants
CDSS user EMR
Opens encounter;
Queries re flu
vaccine;
information sheet
Performance
measure; quality
of care
Primary
prevention
of
influenza
for all
eligible
patients
Medium Additional
task to
remember (but
minimal
increase in
workload)
Office
Manager
Consultant/practic
e management
Orders flu vaccine;
oversees staffing
needs
Performance
measure; quality
of care;
Vaccine
inventory
control;
maintain
patient
flow
High Work
slowdown;
staffing,
vaccine
shortages
'


Stakeholde
r
Role/Participation
in
CDSS
Motivation Goals Stake
in CDS
Concerns
Project
Manager
Consultant/manage
r
Manage CDSS
development,
implementation

Performanc
e measure
for job
(raise,
promotion)
Meet or exceed
budget/timelin
e goals; client
satisfaction
High Staff
resistance to
change;
technical
difficulty;
cost
Systems
Analyst
Consultant/analyst
Workflow analysis;
intervention
specification
Performanc
e measure
for job
(raise,
promotion)
Accurate
assessment of
client needs
Medium Staff
resistance;
understandin
g of
workflow
Systems
Designer
Consultant/designer
System design;
troubleshooting
Performanc
e measure
for job
(raise,
promotion)
System
functions as
required
Medium Technical
challenges;
time/cost
constraints
Trainer/
tester
Consultant
/implementer
Train staff;
oversee go-live;
measure outcomes;
staff support
Performanc
e measure
for job
(raise,
promotion)
Staff use of
CDSS; positive
outcomes
measures
Medium Staff
resistance to
use or
rejection of
CDSS
Table 2. Technical stakeholders

Information System Inventory
This is an inventory of the resources that will be necessary to implement the proposed influenza
vaccination intervention. The technical solution is designed to be integrated within the Electronic
Medical Record and may also interact with a data warehouse and a patient reminder solution.
The inventory is separated into three sections: hardware, software, and personnel.




(


Hardware Inventory

Equipment Purpose Additional Detail
Application Servers Host the applications EMR, CDS, Reminder Solution.
Existing equipment should be
sufficient.
Database Servers Host the databases Data from applications. Existing
equipment should be sufficient.
Web Servers Allow for patients to use the
patient portal via the internet
May also be needed for reminder
solution if using email reminders.
Existing equipment should be
sufficient.
Workstations Allow for users to interact with
the system.
Existing equipment should be
sufficient. Used for viewing and
recording information as well as
entering orders.
Scanners Record documentation from
patients regarding externally
received vaccinations
Existing equipment used for
scanning other patient documents
should be sufficient.

Software Inventory

System or Element Purpose Additional Detail
Electronic Medical Record
Discrete data fields For recording of information
relative to vaccinations
Vaccination type, vaccination
date
Query tools Ability to identify patients within
(or outside of parameters)
Vaccination type, vaccination
date, age range, health status
Data Warehouse (depending on
architecture)
Allow for collection and
querying of data

Computerized Physician Order
Entry System (CPOE)
Allow for entry of vaccination
order

Patient Portal Allow for patients to connect to
their medical record
Receive reminders, receive
education, schedule
appointments, upload
information regarding externally
received vaccinations
Reminder solution Used for automated contacting of
patients
Email, text, phone calls
Electronic learning tools For Education Provider education on tools and
processes. Patient education on
vaccinations and the patient
portal


)

Personnel Inventory

Role Duties Additional Detail
Subject Matter Experts Provide clinical expertise to
development team on process
Physicians, Nurses, Medical
Assistants, Schedulers
Analysts Translate clinical requirements to
technical requirements
Must have both clinical and
technical understanding. Need to
have understanding of tools for
documenting requirements
(Excel, Visio)
Developers Build technical solution Create queries, add discrete fields
to screens/database as needed,
Trainers Develop materials and train users
on solution
Materials may be needed for the
care team and patients. Training
may be needed for the care team
Testers Test solution There will be unit testing,
integrated testing, and user
acceptance testing
Users Use solution. Provide feedback Physicians, Nurses, Medical
Assistants, Schedulers as well as
Patients

Intervention Selection and Workflow Opportunities
Improving Outcomes with Clinical Decision Support: An Implementers Guide suggests four
possible CDS interventions on Pages 165-170 which are applicable this CDSS. Of the four
possibilities, all are relevant in some way, although one would need to be altered to create a
relevant data presentation which recognizes the annual CDC influenza guidelines and
incorporates its data regarding influenza dates, types, and vaccines. The text also makes several
important recommendations on Pages 153-163 regarding the use of multiple interventions, the
empowering of multiple staff members (such as Medical Assistants), and the introduction of
interventions at appropriate points in the workflow.
The proposed decision support is intended for use in the outpatient environment and will
alert providers, patients, and office staff. The first intervention involves the identification of
*

patients without a current vaccination in their EMR. Automated notifications will be provided to
the patients via the patient portal, text message, or voice message. Patients with appointments in
the near future would receive an informational message alerting them they are due for their
vaccination and tell them where more information can be obtained. Patients without upcoming
appointments would be provided an opportunity to schedule an appointment, information
regarding alternate locations for receiving their vaccination, and information on how to learn
more about the vaccination. The system will also provide the patient with an opportunity to
report information regarding vaccinations received outside the health care system.
A typical workflow into which these interventions can be introduced, which shown in
figure one. A medical assistant (MA) places the patient in the exam room, opens the encounter
note, and enters the chief complaint as stated by the patient. The second intervention is
incorporated into the encounter note through an alert that prompts the MA to ask the patient the
date of their most recent influenza vaccination if this information is not in their EMR; the third
intervention will verify that the date entered is after the release date of the annual CDC
guidelines, which are incorporated in the EMR through this CDS and which can be viewed
through a link from the chart to the following website: CDC - Seasonal Influenza (Flu) - Weekly
Report: Influenza Summary Update. The fourth intervention is an alert asking the MA if he
verified that the date of the influenza vaccination occurred this year with the patient. If the
patient requires a vaccination at this visit, the nurse will receive an alert to administer and
document it from this CDS, prompted by the previous entered responses of the MA, before she
closes the note and prior to when the clinician sees the patient; the vaccination will be authorized
through a standing order.
!+

Clinicians receive an alert in the form of the fifth intervention when entering the Plan and
can verify that the patient received the vaccination, discuss any issues regarding the vaccination
with the patient, and offer and order it if necessary. This CDS reminds the provider of the need
for clinician documentation of the vaccination in the EMR, but only during the influenza season,
and alerts the office manager if vaccine shortages occur; indirectly, it also records data on the
efficacy of the system, for the purpose of HHS reporting. The sixth and final CDS intervention is
a dropdown menu from which the clinician can select reasons why the patient refused the
influenza vaccination, if this occurred, for the purposes of data gathering.
Workflows and processes affected will include pre-visit workflows and visit workflows.
Pre-visit changes are in regards to patient identification, patient notification, scheduling, and data
collection. Visit workflow changes include notifications for the medical team, patient education,
and data collection. The lower frequency of immunization is possibly due to general
misperceptions that the flu shot is ineffective, unnecessary, and maybe even dangerous
(Rivers, 2013). In order to change the perception of the effects of the flu shot, patients will likely
have to be repeatedly provided educational material.
Change Management Plan
The proposed influenza vaccination intervention has various effects on many people
during the individual steps in the process. Some effects can be anticipated while others cannot.
The selected methodology to mitigate the risks of the new intervention for influenza vaccination
is comprised of communication and training.
The automatic generation of patient notifications has a direct impact on patients and
indirect effects on the staff. Some patients may appreciate receiving notification of the
!!

recommendation for their vaccination while others may react differently. It is important to
compose patient messages with all the needed content, at the correct reading level, and with the
appropriate level of urgency as to encourage but not scare the patient. Additionally, the clinic
must provide access to additional information and create a methodology for patients to
communicate their feedback. This patient communication potentially creates indirect work for
the clinical staff. They will likely receive questions and potentially receive feedback from
patients who were concerned by the message. Clinical staff will need to be trained regarding how
to answer expected patient questions, where to direct questions they cannot answer, and how to
handle upset patients.
The system provides the ability to record vaccinations occurring outside of the system.
Patients will be given the ability to enter proof of their vaccination via the patient portal. Easy to
understand instructions will be available within the portal, support will have to be available to
patients, and alternative methodologies will also be available (as well as made known to
patients). Secure alternative methods for patients, including the mailing of copies of their
vaccination record or delivery of the record to their physician office, require staff preparation
and training. Staff will be informed of the potential record receipts and taught the proper way to
store the information in the system as well as what to do with the paper. Additionally, staff will
be prepared to receive patient questions regarding the collection of vaccination information.
Finally, a policy will be created for managing patients who have discarded the paperwork from
their vaccination. This policy will be clearly conveyed to both staff and patients.
Staff will receive training regarding changes they will encounter in the electronic medical
record (EMR). Clinical staff in various roles will be trained regarding the alerts they will receive
when seeing a patient in need of a vaccination. They will be educated regarding the meaning of
!#

the alert as well as how to discuss the vaccination with the patient. Additionally, the staff
responsible for administering the vaccination will have to be trained regarding the entry of the
vaccination in the EMR to ensure the entry of quality information in an efficient manner. In the
case of patient denial, staff will be educated regarding how to document the patient denial.

Communication and education will be created at the appropriate level for the intended audiences,
which likely require different materials for patients and staff. Staff communications and
education will be provided prior to system go-live. Patient communication will be provided
with the initial notification so it isnt forgotten during the period between messages. The
organization could conduct free learning workshops for patients, which could also serve as a tool
for enrolling patients in the patient portal, helping the organization achieve meaningful use
measures for patient use of the portal.
System Design
Models
Two models of the CDSS are presented. Figure one models the decision logic for the
system as a flow chart. Figure two models the workflow associated with the patient encounter
showing the user and the point in the visit where each intervention occurs.
!$

Figure 1.
!%

Figure 2. Workflow analysis

!&

Architecture
CDSS architecture covers three aspects: technical design issues, reasoning, and
knowledge acquisition (Berner, 2007). Technical issues include structuring, formatting, and
representing the knowledge-base and the data; reasoning is often rule-based or can be thought of
as one or more of six Bayesian varieties (Berner, 2007). Knowledge is acquired through
ontologies and vocabularies used to capture, enter, name, encode, classify, organize, store,
retrieve, utilize, and manage data (Berner, 2007). The following sections cover each of these
three categories as they pertain to this CDSS.
Knowledge will be multi-sourced and comprehended from data available through the
CDCs websites and also gathered from patient and provider inputs; it will be stored temporarily
on the CDSS office server and permanently within an on-site central repository, where it will be
mined by a data warehousing unit, leading to the generation of reports that will be distributed to
state vaccination registries in their approved formats. Our central medical data repository will be
modeled after Intermountain Health Cares HELP system, with its client-server CDSS and
lifetime data repository upgrades; similarly, it will employ Microsofts OLE DB Provider for
DB2 V3.0 to interface with its Oracle RDBMS V12.1c (Huff, 1994).
The repository collects data from the server each night from which the warehouse is able
to process it, stores its own data separately, and makes it available to end-user interfaces (Szirbik
et al, 2006). The patient portal resembles Partners HealthCares Patient Gateway with its
interactive, algorithm-driven interface, recommendations of what actions should be taken,
(and an) opportunity to check a box or fill in free text to add a concern (that enters) directly into
a physician progress note, thereby facilitating documentation (Grant et al, 2006). It also affords
!'

them the chance to learn more about vaccinations via CDC-based corresponding explanations
provided by a commercial patient education content provider (Grant et al, 2006).
Interfaces will be implemented using Microsoft Internet Information Servers in HTML
format (Grant et al, 2006). The patient will access the EMR via their view, otherwise known as
the patient portal; this EMR is the source for data capture and transfer and the means for sharing
the CDSS among users (Grant et al, 2006). The interface between the on-site and online CDC
databases, and their patient- and provider- users, will employ UMLS tools to conduct searches
through the NLM Gateway system in order to access content at both locations (Lindberg and
Humphreys, 1990). This interface is integrated into both the patient portal and the EMR and
displays as a GUI, and offers the aforementioned drop-down menus and checkboxes (Shiffman,
1994).
Data entry is structured using the items necessary for inclusion such as vaccination type,
date, site of administration, lot number, manufacturer, and vaccinator; similarly, the vaccine
information data is classified by date, source, etc. Contraindications will be cross-checked
against available information on the CDC website and internal database criteria for exclusion;
likewise, vaccination dates will be examined as to whether they fall in the current season. The
algorithms included in this document trigger will be used by the decision logic to trigger the
alerts at each point (Shiffman, 1994).
Rule-based CDSS interaction architecture was also inspired by Partners HealthCare
efforts at representing knowledge. So- called if then statements work well with the paradigm
of our algorithms and workflow diagrams in informing the systems decision logic. The intention
is to minimize data types and create logic expressions (that) can be readily expressed in the
object-oriented GELLO expression language currently under review in HL7 (Greenes et al,
!(

2004). With two applications in operation, the client-server system requires an event monitor,
rule activator, centralized rule base and rule engine, (and) access to data sources for patient
information, as well as to knowledge bases (Greenes et al, 2004). To formulate the rules, we
will follow the steps of identifying and modeling the contexts of the rules, making the system
function in synchronous mode, and having whole data structure mapped into the HL7 RIM
standard data model (Greenes et al, 2004). The event monitor guides the system through
message-driven transactions leading to events/ triggers; actions- orders, approvals, and
cancellations; notifications and recipients (Greenes et al, 2004).
We propose to design a semantic web, derived from a Canadian model, to integrate
vaccination guidelines into the CDSS knowledge base, which establishes ontologies upon which
a rules engine operates. They will be encoded using Protg software and are a CPG-Domain
ontology based on the CDCs recommendations and a Patient ontology based on the EMR
database (Hussain et al, 2007). Like the Canadian model, the former ontology interfaces with the
rules engine which in turn interfaces with the latter to generate the alerts, but the two are also
linked (Hussain et al, 2007).
Intervention Specifications
The proposed clinical decision support system (CDSS) is designed to increase rates of
seasonal influenza vaccination in an ambulatory family practice setting. This will be achieved
through interventions targeted at key persons and occurring at different points in time. This
design will result in multiple opportunities to recommend and administer flu vaccine to patients.
Data capture will allow outcome tracking for quality improvement and to assess the efficacy of
the CDS.
!)

The first intervention will target patients, with a pre-visit reminder delivered via patient
portal. During visits, the remaining interventions will be directed at medical assistants, nurses,
and providers via the electronic medical record, as noted in the figure.
Data entered into the EMR must be converted into coded elements, by the CDS or by
users, when possible. For instance, dates of vaccination can be checked by CDS for proximity to
the recommended vaccination period when they are coded elements. There are three options for
the forms into which data can be converted: CPT procedure codes, ICD-9 procedure codes, and
ICD-9 diagnosis codes relating to influenza and influenza vaccine.
Input will be provided by the EMR, by patients, and by clinical staff. The CDSS can
search for ICD-9 and CPT codes in the EMR. Patients and providers can enter structured data
using template forms with dropdown menus and optional fill-in boxes. Patients will receive
output from the CDSS in the form of texts, emails, and reminders on the secure patient portal.
Medical staff can access the system through the EMR or a PC within the office which is a
terminal for the server in which the CDSS database is stored and from which it operates; the
terminal opens the tracking system for the CDSS which can be used by providers or by IT staff
for software maintenance.
Checklists will be the predominant form of data presentation, for ease of use. Alerts will
appear in a pop-up window and perform the following functions: state the reason for alert;
provide useful, actionable data; list options and/or links for explanations; display choices with
tabs that open checkboxes and/or boxes for data entry and/or exception explanations; and allow
for comments at the conclusion (Osheroff et al, 2011). Alerts will be non-modal so that they do
not impede workflow. The user interface will be standardized with the same general appearance
for ease of use.
!*

All information will be available on the main screen or at most one click away. Text
message, e-mail, and portal websites will be HIPAA-compliant. The CDSS will link directly to
CPOE order checkboxes where providers may edit their selections. The reference display
information will be integrated into the CDSS, EMR, and order lists, such that the CDSS will
search through a reference, pick out the relevant data, and make them coded elements. Clinicians
will be able to override CDSS recommendations in cases where their clinical judgment differs.
Lastly, the CDSS will list exceptions and offer an "other" option when appropriate.
Our system will utilize data-driven triggers to move from one intervention to the next.
Concerning system background logic, the CDSS will utilize the basic alert sequence of data
recognition, triggering, CDSS logic to determine the need for system activation, user
notification, user acknowledgement, data presentation, and user action providing additional data
to be recognized (Osheroff et al, 2012). The CDSS will also screen alerts and their frequency to
ensure that they are displayed only at critical times, to minimize alert fatigue. Figure Two
contains information describing each individual workflow step.
User Interface
The influenza vaccination clinical decision support system has two primary sets of users;
clinicians and patients. Each set of users has a different primary interface. The information
entered and viewed in the primary interface will also be available to the users in other
appropriate places.
Clinicians will enter data and receive feedback within the framework of their standard
Electronic Medical Record (EMR) interface. Alerts will be available on the provider schedule
and the first screen of the patient record during an encounter as well as various appropriate
places within the EMR. A vaccination tab will be available within the patient record as can be
#+

seen in the following mockup. The mockup below is Dr. Jones (our clinician) view of the
vaccination tab for our patient Willie the Wildcat, the Northwestern University Mascot.


The EMR header, including the provider name, links to some frequently used
functionalities, and navigation tabs, are standard across screens within the medical record. This
uniform layout is important in the user interface to provide a consistent feel for the user. Also,
the patients picture, medical record, number, and date of birth are standard across tabs. This
provides additional visual clues to the user that the correct medical record is being used.
The vaccination tab itself has four sections. There is an alert section on the top right of the
screen which will display in red if the patient has a vaccination which should be addressed in the
current visit. In this example, the section header is red as there is an alert since Willie is due for
his influenza vaccination. The section below that is for vaccination history. It is an informational
#!

section displaying the patients most recent vaccinations including when and by whom they were
administered. More information on each vaccination can be accessed by clicking the entry. A full
list of the patients vaccinations can be accessed by clicking the link at the bottom of the section.
The section intended to get the most use is placed at the top center. This is the new vaccination
administration section. The purpose of this section is to record the information regarding the
vaccine administered during the visit. Dropdown boxes are used as possible in order to ensure
quick and accurate data collection. A comment field is provided in case the provider needs to
include additional data not accommodated by the form. Should the patient deny the vaccination,
the final section of the tab is the patient vaccine denial section. This allows the clinician to record
when the vaccine was offered, record education provided, the patients reason for refusal as well
as any additional comments. The tab is uncluttered which should make it easy for clinicians to
navigate while still providing access to all the information they need to enter and view.
Reports regarding the patient populations influenza vaccination will be available to
clinicians via the EMR reporting tools. Available information will include what percentage of
patients has received vaccinations, what percentage has denied vaccinations, what percentage of
patients without vaccinations have a visit scheduled during the vaccination period, and the
percentage that do not. More specific information will be available regarding the types of
vaccination received, patient reasons for denial, and the demographics of those receiving, yet to
receive, or denying a vaccination. These reports will be available at a patient level, a provider
level and at a practice level. This will allow for evaluation from various aspects.
Ideally, patients will receive notification and enter data within the framework of the patient
portal. If the patient has not enrolled in the patient portal, they will be notified of the
recommended vaccination via appropriate means as selected by the patient during a previous
##

visit. Patients will be able to provide a contact preference between email, telephone, text, or
traditional mail. This information will be recorded with their demographics. The EMR will be
integrated with a reminder system, which will create and send the reminder messages to the
patient via the means they have designated. In addition to the vaccination information, reminders
will include information about the patient portal.
Patients who have enrolled in the patient portal will receive an email informing them of a
notification in the portal. When they logon to the portal they will be alerted that they are due for
a vaccination with a link to the vaccination tab of their record. The mockup below is Willies
(our mascot patient) view of the vaccination tab in his patient portal.
#$


The basic layout of the patient portal is very similar to that of the clinician EMR view. This
is because the format is easily navigated by both patients and physicians. Additionally, the
commonalities make it easier for the developers, support desk staff, and others involved with the
EMR by minimizing the variations they have to learn. Therefore, just like in the clinician view,
the EMR header, including the patient name, links to some frequently-used functionalities, and
navigation tabs are standard across screens within the patient view of the medical record. Also
like the clinician view, the patients picture, medical record, number, and date of birth are
standard across tabs. This provides additional visual clues to the user that the correct medical
#%

record is being used. This is especially important for a patient who also manages the medical
treatment of additional family members. The patient has the ability to switch between allowed
family members charts as well.
The patient view of the vaccination tab has sections similar to the four viewed by the
physician plus two additional tabs. The patient alerts section works the same as it does for the
provider with one additional function. If the patient has an active alert, a link will be provided so
the patient can make an appointment. The patient vaccination history also works exactly the
same as it does for the clinician with one additional function. The patient has the ability to print
their vaccination record should they need it for their employer, their childs school, or any other
reason. The I Already Got My Flu Shot section takes the top center place where the clinicians
new vaccination administration section is located. The purpose of this section is to record the
information regarding a vaccine administered outside of the clinic. Just like the provider
functionality, dropdown boxes are used as possible in order to ensure quick and accurate data
collection. Should the patient choose not to obtain the vaccination, the final section of the tab is
the I Dont Plan to Get a Flu Shot section. This is similar to the clinicians patient vaccine
denial section and allows the patient to record their reason for refusal as well as any other
comments. A link to educational materials is also provided for the patient. The two additional
sections are frequently asked questions and a map displaying vaccination locations near the
patient. The tab is designed to be easy for patients to use in order to encourage the entry of
information.
The influenza vaccination clinical decision support system can run autonomously for
standard annual vaccinations. However, if there are any changes to the recommended dates or
unscheduled vaccination recommendations, they would have to be entered manually. Therefore,
#&

an interface will be designed for administrators to help ensure the staff and patients do not
receive alerts prior to the vaccination being available. The updates could be entered globally,
meaning one entry could be applied to all patients. The long term goal is for the CDS to run
independently. In order for this to occur, the system will have to be able to receive vaccine
recommendations from the Center for Disease Control (CDC) and/or other entities. Until that
interface can be created, the system will require human entry.
The system is able to act autonomously to evaluate the patient records to determine if they
have a vaccination recorded during the set period. It can create alerts and reports, remove alerts
and update reports dynamically as information regarding completed vaccinations is entered. The
ability to enter denial of the vaccination allows the patient to opt out of additional notifications,
but will only change the providers view to display the patient has chosen not to receive the
vaccination rather than remove it entirely. This method will allow the clinician to be informed
and to choose whether or not to make an additional attempt to recommend the vaccination to the
patient.
Knowledge Engineering
The primary source of information and logic for the CDSS is the Center for Disease
Control (CDC) Advisory Committee on Immunization Practices (ACIP) recommendations for
the prevention and control of influenza, which is revised annually prior to each flu season (ACIP,
2012). ACIP is a multidisciplinary group that includes representatives from internal medicine,
pediatrics, epidemiology, infectious diseases and other relevant medical specialties (CDC, 2013).
This body is widely accepted as the gold standard reference source where immunization issues
are concerned. The recommendation for the 2013-2014 flu season is that all persons older than 6
months be screened for eligibility to receive flu vaccine and, if indicated, receive the vaccine as
#'

early as possible in the flu season, which occurs each year between October and March.
Contraindications to influenza vaccine are relative and should be considered on a case-by-case
basis (ACIP, 2012). The CDSS includes a link to the CDC Fluview web page, which provides
extensive, high-quality information for patients and providers about influenza, including
surveillance of disease spread during flu season (CDC, 2013).
The clinical outcome measure on which the CDSS is evaluated is based on the National
Quality Forum performance measure for influenza immunization (ID #IMM-2) (NQF, 2012).
The numerator is the number of patients older than 6 months who are screened and vaccinated, if
indicated. The denominator is the number of patients over 6 months of age in the practice. In
addition to comparing immunization rates within the practice before and after the CDSS, practice
rates will be compared with national average rates of influenza immunization tracked by the
CDC.
The Unified Medical Language System (UMLS) is the source for ontological
representations in the CDSS. Specifically, SNOMED-CT is the basis for terminology relating to
influenza and related disease concepts; the International Classification of Disease (ICD-9-CM)
classification system is used for diagnosis codes and billing; and the Current Procedural
Terminology (CPT) coding system is used for procedural codes for documentation and billing.
(Berner, 2007).
The knowledge content of the CDSS will be deployed in the forms of patient reminders,
links to information and frequently asked questions through the patient portal, patient education
video material, EMR alerts to clinicians, and vaccine order sets and documentation templates.
Clinicians will be able to link from the EMR with one click to the CDC Fluview webpage for up-
to-the-minute information regarding the current flu season. Information available on Fluview that
#(

clinicians would find useful at the point of care includes contraindications to vaccination, types
and brands of influenza vaccine, dosage and administration guidelines, and recommendations
and cautions for children and adults of different ages. (CDC, 2013).
The decision to give the influenza vaccination is straightforward for the clinician: the
vaccine is safe and effective, with minimal side effects, and is recommended for all individuals
older than 6 months (CDC, 2013). In view of this, advanced algorithms such as those employing
Bayesian logic are not indicated. A procedural knowledge representation using the ifthen
type as the primary logic rule is appropriate for this CDSS, although some Boolean logic will
also be applied (Berner, 2007). For example, the patient reminder to get a flu shot would be
triggered in the following sequence: IF date is after Sept 1 AND before March 30, THEN the
system searches EMR for evidence of flu vaccination. IF no vaccine found, THEN notification is
sent to patient. IF a patient indicates by clicking a button that they want the flu shot, THEN the
system is triggered to go to the scheduling page.
Management of knowledge assets are outlined in a knowledge maintenance policy and are
the responsibility of Dr. Smith, a project champion and the primary liaison with the
implementation team (Osheroff, 2012). Using automated reports generated monthly by the
EMRS, along with qualitative input from interviews and surveys of the other users, Dr. Smith
tracks measurable outcomes as well as subjective feedback on the knowledge content of the
CDSS. Each September, Dr. Smith and the other providers review ACIP guidelines for the
upcoming flu season and make adjustments as needed to keep CDSS interventions in line with
best practices. Planning for the anticipated transition from ICD-9-CM to ICD-10-CM is
underway. Similarly, changes to SNOMED-CT and CPT will need to be incorporated. Hardware
#)

and software updates or changes should be followed by testing of the CDSS to ensure its
knowledge content assets function as intended.
System Evaluation
In order to monitor usability, multiple methods will be used. Page hits will be monitored.
This will allow for the understanding of what pages are visited and how often. It is important to
understand how many unique patients visit the page and how many of those have either
scheduled an appointment, recorded the occurrence of a vaccination, or recorded the denial of
vaccination. If there is a large group of visitors that does not take any of these actions, the
usability should be revisited. Additionally, the vaccination tab will be included in periodic user
group meetings for the EMR and the patient portal. These meetings should occur for all
functionality within those systems. Finally, a feedback form will be available on both the
clinician and the patient pages.
Based on a review of literature, there are several additional techniques we will employ to
determine the efficacy of our system design, its appropriateness to the clinical problem we are
attempting to solve, and how effective it is in performing its intended functions. In addition to
conducting these tasks, we will compare it to a similar system in order to assess our capabilities
as well as those of our CDSS. The first step in the evaluation of our CDSS is to verify how well
this construct meets the specifications we proposed in order to achieve our stated goals. Initially,
we will conduct an analysis of patient and provider opinions, by surveying their perceptions of
the systems utility and consistency, of the systems usability and quality, and of their
willingness to use it (Hwang et al, 2004). Next, we will evaluate our implementation methods
and process outcomes using data analysis tools to monitor whether alerts are positioned at the
correct points in the workflow, whether they should require responses or not, whether they
#*

directed providers to act and act correctly, and whether the data sources we have referenced are
accurate (Bryan and Boren, 2008). Finally, we will attempt to confirm our choices of CDS alert
types and discuss ways we could improve the system such as the possible use of decision trees or
Bayes theorem (Tenorio et al, 2011).
Methods we will employ to validate the use of the system are the following. Since the
primary purpose of this project is to increase vaccination compliance, we will use the previous
years statistics, prior to the CDSS implementation, as our control group. Depending on how low
or high the national averages are for a calendar year, we can set reasonable goals but also
anticipate that as our vaccination percentages increase, we expect a slowing in improvement over
time. As was mentioned earlier, we will evaluate the patients receiving alerts, either directly or
at a provider visit, and determine which ones received vaccinations, to judge the impact of the
project.
To judge the systems efficacy, we will begin by quantifying patient and provider
responses to the alerts and will follow that up by tracking outcomes, namely the intended
increase in patient vaccinations. One way to achieve these measurements would be to set up a
randomized controlled trial with the alert being the intervention, administration/ documentation/
ordering of the vaccination being the variables measured, and intent-to-treat being the analysis of
the data performed (Boustani et al, 2012). The ethical issue this creates would be that we would
need to ensure that those patients for whom the providers were not alerted by the CDSS still
received their shots, so this may not be feasible. The next evaluation would be to judge the
impact of the CDSS on workflows, considering the concomitant rise of flu-like illnesses in the
Fall. Receiving the vaccination during a visit should enhance workflows and patient satisfaction
by reducing the number of visits, both of which we can monitor (Manotti et al, 2005).
$+

To know whether or not our alerting method is beneficial, we could compare our design to
others, in terms of compliance rates; although we intend to use pop-ups, interactive alerts or
pharmacy interventions are viable alternatives and may work well in the case of a patients visit
to a pharmacy to receive their prescriptions (Scheepers-Hoeks, 2013). We also need to determine
whether the alerts are having the same effect on each subset of provider staff; using time stamps,
we could graph when the shots are administered to further enhance workflows (Mann et al,
2011). We could also survey providers regarding their opinions on universal vaccination and the
necessity of offering it; this is especially crucial in the cases of patients who are non-compliant
or refuse to receive the vaccination (Robbins et al, 2012). This could lead to the development of
incentives for CDSS use. We would also hope to see an increased rate of vaccination among
patients with chronic diseases who are susceptible to influenza and its complications, so we
could analyze data in this group separately (Nendaz et al, 2010). Implementation can be
evaluated by measuring training hours and syllabi; subtle changes in graphics or colors- or in the
duration of the alert- could be made to screen for their effects on compliance; and the effects of
alerting multiple employees could also be scrutinized (Scheepers-Hoeks, 2013).
There is a real-world version of our product which meets the same needs we observed,
but which was customized to cover a larger group of patients across a health system and
vaccinations covering other disease conditions (Swenson et al, 2012). It functions similarly to
ours in that it determines who is eligible for immunization and then alerts providers to act, or
indicates that they refused it; the developers of this system also saw the need to make these
duties those of MAs, just as we concluded, and to integrate it into the workflow (Swenson et al,
2012). Even their assessment criteria, i.e. quantifying immunization rates, correspond with ours;
with its success rates such as a 10% increase in immunization rates, its similar ability to
$!

incorporate new indications, and its room for expansion, this study reassures our team that we
can expect good results as well (Swenson et al, 2012). Our goal is to provide additional data
demonstrating the potential that this intervention has on improving care quality.
Discussion
This report describes the planning, design, and evaluation of a CDSS in a family practice
setting. While the overarching goal of the CDSS is to support universal vaccination (CDC,
2010) of all eligible patients in the practice, the specific, measurable outcome is a significant
increase in influenza vaccination rates following CDSS implementation and compared with rates
in previous years. Review of the literature shows that nationwide influenza vaccination rates
have been under 50 percent, despite strong recommendations and coordinated efforts at the
federal, state and local levels to improve compliance (CDC, 2013).
The framework for this intervention package is the CDS Five Rights: to achieve the
desired objective by delivering the right information to the right person in the right format,
through the right channel and at the right point in the workflow. (Osheroff, 2012) This CDSS is
designed to deliver information to these key persons along the chain of events resulting in
seasonal influenza vaccination:
The Patient is notified that flu season has begun and is prompted to make an
appointment for vaccination.
The Medical Assistant the first clinical contact when the patient arrives at the office
is prompted to ask the patient whether they have had the flu shot, and if they wish to get
one.
The Nurse is the person who administers and documents the vaccination.
$#

The Provider is the person responsible for answering patient questions and concerns
about the safety and efficacy of the influenza vaccine, for checking that each patient is
an appropriate candidate for vaccination, and for ordering the appropriate type and
dosage of vaccine.
Planning of the CDSS implementation was undertaken as a team effort involving technical
and clinical stakeholders at each phase of development. A complete information system
inventory and iterative workflow analysis formed the foundation for the CDSS package of six
interventions targeting each of the users at different points in time. Change management analysis
and planning aimed to anticipate the effects of the CDSS on users and workflows, with the goal
of targeting training and communication efforts to minimize disruption.
System design began with the two models developed during the planning stage. System
architecture is based on the stated requirements of the practice owners and takes into account
their technical and personnel resources. Intervention specifications are based on the CDS Core
Actions of pattern recognition (recognizing flu season by date); formulating a plan (prompting
patient to make appointment); executing the plan (vaccine order sets); monitoring and
responding (checking for current vaccine and alerting); and communicating (patient reminders)
(Osheroff, 2012). Interventions are designed for ease of use, with non-modal alerts, checklists,
actionable data, and links to action steps. Specification for collecting data on reasons for vaccine
refusal is included for system evaluation and ongoing quality improvement efforts. Graphical
interfaces of the CDSS are designed to facilitate intuitive use and to take into account the
intended users: interventions directed at patients are configured at appropriate reading level, with
medical content presented in terms understandable to laypersons (Berner, 2007), while EMR
interventions for clinicians are configured for efficient, non-disruptive alerting and integration
$$

into workflows. Knowledge engineering for the CDSS includes identification of the information
sources for the interventions and outcome measurement, procedural representation of knowledge
content in a variety of interventions, and planning for knowledge content maintenance.
Evaluation of influenza vaccination rates before and after CDSS implementation is the
primary, but not the sole, outcome measure. Other assessments will consider the efficacy and
usability of the individual interventions, and the adequacy of training. A combination of
subjective and objective evaluation will yield the most useful information for updating and
refining the CDSS; an iterative evaluation process beginning during development and continuing
through implementation is preferred (Horsky, 2010).
While the CDSS described here has been configured according to many principles of
effective CDS design (Osheroff, 2012), there are several assumptions worth mentioning. First,
the design assumes no cost barriers, which is unlikely to be the case regardless of the client, but
especially in a private-practice setting. This could be addressed by including a project budget and
cost-benefit analysis in the planning phase. Another assumption is that the recommendation for
universal influenza vaccination is unlikely to be rescinded or substantively changed for the
foreseeable future of the CDSS. This is a fair assumption, based on the well-established safety
and efficacy of influenza vaccination. However, CDS design and maintenance must take into
consideration that medical knowledge is changing constantly and sometimes quite dramatically;
an outdated system that recommends inappropriate actions is more than useless it is a liability.
Having a knowledge maintenance policy and plan helps to avoid this pitfall. Another assumption
is that all patients will have access to the patient portal in order to receive and act on flu
reminders. The options of telephone, mail, email and text message reminders have been included
for those who are unable to access the portal. For reasons of clarity, the scope of the CDSS was
$%

limited to the routine immunization of patients without complicating factors or contraindications.
Future extension of the model could include an algorithm for patients with egg allergy, a relative
contraindication to influenza immunization; and for pediatric patients, for whom dosing and
administration are more complex (CDC, 2013). Use-case scenarios such as these should be
considered at the outset of the development of CDSS, as well as during implementation and
evaluation (Osheroff, 2012).
Despite ongoing clinical trials, there remains no consensus that current CDSS improve
patient outcomes and quality of care (Garg, 2005). The 2010 report, Clinical decision support:
progress and opportunities, cites three strategic objectives to advance the state of CDS: 1) best
knowledge available when needed; 2) high adoption and effective use; and 3) continuous
improvement of knowledge and CDS methods (Lyman, 2010). While larger trials are needed to
validate the use of CDS, the increasingly widespread use of EMR in medical practice offers
myriad opportunities to incorporate well-designed CDS into daily workflows, as described in this
report, as a means of supporting the highest standards of care.

References
Advisory Committee on Immunization Practices (ACIP). Prevention and control of
influenza with vaccines: Recommendations of the Advisory Committee on Immunization
Practices (ACIP) United States, 2012-13 influenza season. MMWR. 2012; 61(32): 613-618.
Bates, DW, Kuperman GJ. Ten Commandments for effective clinical decision support:
Making the practice of evidence-based medicine a reality. Journal of the American Medical
Informatics Association. 2003:10(6); 523-530.
$&

Berner, H.S. (Ed.) (2007). Clinical decision support systems: Theory and practice (2nd
ed.). New York, New York: Springer, Health Informatics Series.
Boustani, M. A., Khan, B. A., Abernathy, G., Zawahiri, M., Campbell, T., Buckley, J. D.,
& Callahan, C. M. (2012). Enhancing care for hospitalized older adults with cognitive
impairment: a randomized controlled trial. Journal of general internal medicine, 27(5), 561-567.
Bryan, C., & Boren, S. A. (2008). The use and effectiveness of electronic clinical decision
support tools in the ambulatory/primary care setting: a systematic review of the
literature. Informatics in primary care, 16(2), 79-91
Centers for Disease Control and Prevention (CDC). (2013). ACIP member roster: July 1,
2013 - June 30, 2014. Downloaded from http://www.cdc.gov/vaccines/acip/committee/members-
archive.html.
Centers for Disease Control (CDC). 2013. Recommendations of the Advisory Committee
on Immunization Practices (ACIP). Retrieved from www.cdc.gov.
Centers for Disease Control and Prevention (CDC). (2013). What You Should Know for
the 2013-2014 Influenza Season. Retrieved from http://www.cdc.gov/flu.
Center for Medicare and Medicaid Services (CMS). (2012). Meaningful Use. Retrieved
from www.cms.gov.
Dexter PR, Perkins S, Maharry K, et al. Inpatient computer-based standing orders vs
physician reminders to increase influenza and pneumococcal vaccination rates: A randomized
trial. JAMA. 2004; 292(19):2366-2371.
Fiks AG, Grundmeier RW, et al. Impact of clinical alerts within an electronic health record
on routine childhood immunization in an urban pediatric population. Pediatrics. 2007:120(4);
707-714.
$'

Fiks AG, Hunter KF, et al. Impact of electronic health record-based alerts on influenza
vaccination for children with asthma. Pediatrics. 2009:124(1);159-169.
Garg, A. X., Adhikari, N. K., McDonald, H., Rosas-Arellano, M. P., Devereaux, P. J.,
Beyene, J., & Haynes, R. B. (2005). Effects of computerized clinical decision support systems on
practitioner performance and patient outcomes: A systematic review. JAMA: the journal of the
American Medical Association, 293(10), 1223-1238.
Grant, R. W., Wald, J. S., Poon, E. G., Schnipper, J. L., Gandhi, T. K., Volk, L. A., &
Middleton, B. (2006). Design and implementation of a web-based patient portal linked to an
ambulatory care electronic health record: patient gateway for diabetes collaborative
care. Diabetes technology & therapeutics, 8(5), 576-586.
Greenes, R. A., Sordo, M., Zaccagnini, D., Meyer, M., & Kuperman, G. J. (2004). Design
of a standards-based external rules engine for decision support in a variety of application
contexts: report of a feasibility study at Partners HealthCare System. Medinfo, 2004, 611-615.
Haueis, P., Greil, W., Huber, M., Grohmann, R., Kullak-Ublick, G. A., & Russmann, S.
(2011). Evaluation of drug interactions in a large sample of psychiatric inpatients: a data
interface for mass analysis with clinical decision support software. Clinical Pharmacology &
Therapeutics, 90(4), 588-596.
Horsky, J., McColgan, K., Pang, J.E., Melnikas, A.J., Linder, J.A., et al. (2010).
Complementary methods of system usability evaluation: Surveys and observations during
software design and development cycles. Journal of Biomedical Informatics, 43, 782-790.
Huff, S. M., Haug, P. J., Stevens, L. E., Dupont, R. C., & Pryor, T. A. (1994). HELP the
next generation: a new client-server architecture. In Proceedings of the Annual Symposium on
Computer Application in Medical Care (p. 271). American Medical Informatics Association.
$(

Hussain, S., Abidi, S. R., & Abidi, S. S. R. (2007). Semantic web framework for
knowledge-centric clinical decision support systems. In Artificial intelligence in medicine (pp.
451-455). Springer Berlin Heidelberg.
Hwang, H. G., Chang, I. C., Hung, W. F., Sung, M. L., & Yen, D. (2004). The design and
evaluation of clinical decision support systems in the area of pharmacokinetics. Informatics for
Health and Social Care, 29(3-4), 239-251.
Lindberg, D. A., & Humphreys, B. L. (1990, November). Concepts, Issues, and Standards.
Current Status of the NLM's Umls Project: The UMLS Knowledge Sources: Tools for Building
Better User Interfaces. In Proceedings of the Annual Symposium on Computer Application in
Medical Care (p. 121). American Medical Informatics Association.
Lorenzi, N. M. and Riley, R.T. (2000 Mar/Apr) Managing Change: An Overview. Journal
of the American Medical Informatics Association, Vol. 7 No. 2 pp 116-124
Lyman JA, Cohn WF, Bloomrosen M, et al. Clinical decision support: progress and
opportunities. JAMIA. 2010; 17:487-492.
Mann, E. A., Jones, J. A., Wolf, S. E., & Wade, C. E. (2011). Computer decision support
software safely improves glycemic control in the burn intensive care unit: a randomized
controlled clinical study. Journal of burn care & research: official publication of the American
Burn Association, 32(2), 246.
Manotti, C., Pattacini, C., Quintavalla, R., Tagliaferri, A., Lombardi, M., & Tassoni, M.
(2005). Computer assisted anticoagulant therapy. Pathophysiology of Haemostasis and
Thrombosis, 33(5-6), 366-372.
Medicare Preventive Services. (2012). Quick reference information: Medicare
Immunization Billing. Retrieved from www.cms.gov.
$)

National Quality Forum. (2012). Specifications manual for national hospital inpatient
quality measures, version 4.0a. Downloaded from http://www.qualityforum.org.
Nendaz, M. R., Chopard, P., Lovis, C., Kucher, N., Asmis, L. M., Drffler, J., &
Bounameaux, H. (2010). Adequacy of venous thromboprophylaxis in acutely ill medical patients
(IMPART): multisite comparison of different clinical decision support systems. Journal of
Thrombosis and Haemostasis, 8(6), 1230-1234.
O'Reilly, D., Holbrook, A., Blackhouse, G., Troyan, S., & Goeree, R. (2012). Cost-
effectiveness of a shared computerized decision support system for diabetes linked to electronic
medical records. Journal of the American Medical Informatics Association, 19(3), 341-345.
Osheroff JA, Teich JM, et al. (2012). Improving Outcomes with Clinical Decision Support:
An implementer's guide. Chicago, IL: HIMSS.
Rivers, Caitlin. "The Public Health Paradox: Why People Don't Get Flu Shots - Caitlin
Rivers." Caitlin Rivers. N.p., 28 Jan. 2013. Web. 03 Aug. 2013.
Robbins, G. K., Lester, W., Johnson, K. L., Chang, Y., Estey, G., Surrao, D., & Freedberg,
K. A. (2012). Efficacy of a Clinical Decision-Support System in an HIV PracticeA Randomized
Trial. Annals of internal medicine, 157(11), 757-766.
Scheepers-Hoeks, A. M. J., Grouls, R. J., Neef, C., Ackerman, E. W., & Korsten, E. H.
(2013). Physicians responses to clinical decision support on an intensive care unitComparison
of four different alerting methods. Artificial intelligence in medicine.
Shiffman, R. N. (1994). Towards effective implementation of a pediatric asthma guideline:
integration of decision support and clinical workflow support. InProceedings of the Annual
Symposium on Computer Application in Medical Care (p. 797). American Medical Informatics
Association.
$*

Stojadinovic, A., Eberhardt, J. S., Ward, E. B., Nissan, A., Johnson, E. K., Protic, M., &
Steele, S. R. (2013). Clinical Decision Support and Individualized Prediction of Survival in
Colon Cancer: Bayesian Belief Network Model. Annals of surgical oncology, 20(1), 161-174.
Swenson, C. J., Appel, A., Sheehan, M., Hammer, A., Fenner, Z., Phibbs, S., & Main, D. S.
(2012). Using information technology to improve adult immunization delivery in an integrated
urban health system. Joint Commission Journal on Quality and Patient Safety, 38(1), 15-23.
Szirbik, N. B., Pelletier, C., & Chaussalet, T. (2006). Six methodological steps to build
medical data warehouses for research. International Journal of Medical Informatics, 75(9), 683-
691.
Tenrio, J. M., Hummel, A. D., Cohrs, F. M., Sdepanian, V. L., Pisa, I. T., & de Ftima
Marin, H. (2011). Artificial intelligence techniques applied to the development of a decision
support system for diagnosing celiac disease. International Journal of Medical
Informatics, 80(11), 793-802.