Vol. 49 No.

4 April 2015 Journal of Pain and Symptom Management 773

Special Article

Measuring What Matters: Top-Ranked Quality Indicators for Hospice

and Palliative Care From the American Academy of Hospice and
Palliative Medicine and Hospice and Palliative Nurses Association
Sydney Morss Dy, MD, MSc, Kasey B. Kiley, MPH, Katherine Ast, MSW, LCSW, Dale Lupu, PhD,
Sally A. Norton, PhD, RN, FAAN, Susan C. McMillan, PhD, ARNP, FAAN, Keela Herr, PhD, RN, AGSF, FAAN,
Joseph D. Rotella, MD, MBA, FAAHPM, and David J. Casarett, MD, MA
Johns Hopkins Sidney Kimmel Cancer Center (S.M.D.), Baltimore, Maryland; Johns Hopkins Hospital (K.B.K.), Baltimore, Maryland;
American Academy of Hospice and Palliative Medicine (K.A., D.L.), Chicago, Illinois; University of Rochester School of Nursing (S.A.N.),
Rochester, New York; University of South Florida College of Nursing (S.C.M.), Tampa, Florida; University of Iowa College of Nursing (K.H.),
Iowa City, Iowa; Hosparus (J.D.R.), Louisville, Kentucky; and University of Pennsylvania Perelman School of Medicine (D.J.C.),
Philadelphia, Pennsylvania, USA

Abstract
Context. Measuring quality of hospice and palliative care is critical for evaluating and improving care, but no standard U.S.
quality indicator set exists.
Objectives. The Measuring What Matters (MWM) project aimed to recommend a concise portfolio of valid, clinically
relevant, cross-cutting indicators for internal measurement of hospice and palliative care.
Methods. The MWM process was a sequential consensus project of the American Academy of Hospice and Palliative
Medicine (AAHPM) and Hospice and Palliative Nurses Association (HPNA). We identified candidate indicators mapped to
National Consensus Project (NCP) Palliative Care Guidelines domains. We narrowed the list through a modified Delphi rating
process by a Technical Advisory Panel and Clinical User Panel and ratings from AAHPM and HPNA membership and key
organizations.
Results. We narrowed the initial 75 indicators to a final list of 10. These include one in the NCP domain Structure and
Process (Comprehensive Assessment), three in Physical Aspects (Screening for Physical Symptoms, Pain Treatment, and
Dyspnea Screening and Management), one in Psychological and Psychiatric Aspects (Discussion of Emotional or
Psychological Needs), one in Spiritual and Existential Aspects (Discussion of Spiritual/Religious Concerns), and three in
Ethical and Legal Aspects (Documentation of Surrogate, Treatment Preferences, and Care Consistency with Documented
Care Preferences). The list also recommends a global indicator of patient/family perceptions of care, but does not endorse a
specific survey instrument.
Conclusion. This consensus set of hospice and palliative care quality indicators is a foundation for standard, valid internal
quality measurement for U.S. settings. Further development will assemble implementation tools for quality measurement and
benchmarking. J Pain Symptom Manage 2015;49:773e781. 2015 American Academy of Hospice and Palliative Medicine.
Published by Elsevier Inc. All rights reserved.

Key Words
Palliative care, pain measurement, hospice care, quality of health care, quality indicators, advance care planning, patient satisfaction

Address correspondence to: Sydney Morss Dy, MD, MSc, Johns Accepted for publication: January 21, 2015.
Hopkins Sidney Kimmel Cancer Center, Room 609, 624 N.
Broadway, Baltimore, MD 21205, USA. E-mail: sdy@
jhsph.edu

2015 American Academy of Hospice and Palliative Medicine. 0885-3924/$ - see front matter
Published by Elsevier Inc. All rights reserved. http://dx.doi.org/10.1016/j.jpainsymman.2015.01.012
774 Dy et al.
Introduction
Far too many patients and families in the U.S. expe-
rience unnecessary physical and emotional suffering
during serious and life-threatening illnesses.1 Recent
studies have demonstrated gaps in quality of care in
domains such as pain and symptom management,
communication, and care planning across settings
such as hospitals, nursing homes, and ambulatory
care.2e4 Palliative care is defined as care that provides
relief from symptoms and supports quality of life for
patients with serious advanced illness and their fam-
ilies, and hospice care is an approach focused on pa-
tients with limited life expectancy and their
families.1 Accumulating research demonstrates that
interventions such as ambulatory palliative care
clinics, structured goals of care discussions in critical
care, and outpatient nurse-led interventions targeting
patient/family pain management can improve out-
comes such as patient and family satisfaction and
health care utilization.5
Measuring the quality of care delivery is integral to
hospice and palliative care programs, as specified by
the National Consensus Project (NCP) Clinical Prac-
tice Guidelines and the Joint Commission Advanced
Certification Program for Palliative Care6 and the Cen-
ters for Medicare & Medicaid Services (CMS) for the
hospice programs.7 A critical first step in improving
care is demonstrating where quality deficits exist
compared with national benchmarks and determining
where quality improvement initiatives are most
needed and most likely to be beneficial, but no na-
tional U.S. (or universal) standards for measurement
or databases that would support benchmarking yet
exist. Quality indicator sets for a variety of relevant
populations and settings have been developed and
tested8e11 and the National Quality Forum (NQF)
has endorsed a group of palliative care indicators11
suitable for accountability, such as public reporting.
However, many of the indicators intended for account-
ability may not be the most appropriate for internal
use in clinical programs, and a standard, concise,
cross-cutting set of indicators that can be used inter-
nally for benchmarking, comparison across programs,
and quality improvement is needed. Many palliative
care programs are not yet routinely measuring quality,
many are using locally developed, non-validated indi-
cators, and for those who wish to measure quality,
there are no nationally used sets with benchmarking
that apply across populations and settings.
The Quality and Practice Standards Committee of
the American Academy of Hospice and Palliative Med-
icine (AAHPM), therefore, initiated the Measuring
What Matters (MWM) consensus project, which was
joined by the Hospice and Palliative Nurses Associa-
tion (HPNA) Research Advisory Council. The overall
Vol. 49 No. 4 April 2015
goal of the project was to recommend a concise port-
folio of valid, clinically relevant, cross-cutting perfor-
mance indicators for internal measurement for
hospice and palliative care programs. The intent was
to develop a common core set from which programs
could select, to help create standards for quality mea-
surement of palliative care in the U.S. and allow for
national benchmarking. The aspirational goal was an
initial set of process and outcome indicators that apply
regardless of diagnosis, organizational structure, or
setting, although the MWM team recognized that
such indicators may not always be appropriate or not
yet exist and that some included indicators as
currently defined may not apply across populations
and settings. The MWM team acknowledged that pro-
moting high-quality hospice and palliative care indica-
tors for accountability with CMS and other groups is
also an important goal,1 but may be less relevant for
program development or for improving the patient
and family experience of care; this was, therefore,
not a goal of the MWM project. In this article, we
describe the MWM methodology and the initial core
consensus set of quality indicators.
Methods
MWM was a sequential consensus project, directed
by a partnership between AAHPM and HPNA with a
modified Delphi rating process by first a Technical
Advisory Panel (TAP) and then a Clinical User Panel
(CUP), followed by input solicited from AAHPM and
HPNA membership and from external organizations
and patient advocacy groups to obtain the final indica-
tor set (Table 1). The project goals described above
guided the MWM process. In particular, the MWM
team aligned the set with the NCP guidelines,
including organizing by the eight domains, and other
existing quality initiatives whenever possible. The
team envisioned the indicator set for population-
level measurement, and it may be insufficient for mea-
surement targeting specific settings or populations.
The MWM process also evaluated whether denomina-
tors of existing indicators should be considered for
future modification to be more inclusive and cross-
cutting across settings or populations.
Identifying Indicators
The MWM project began by identifying existing
U.S. process and outcome indicators relevant to hos-
pice and palliative care, available in the public domain
as of October 2013 and specified for U.S. data sources
and developed through a rigorous process and/or
tested for reliability and validity in English. Sources
included indicators endorsed by the NQF and/or in
Vol. 49 No. 4 April 2015 MWM in Hospice and Palliative Care
Table 1
Steps in the Measuring What Matters (MWM) Process for
Selecting the Initial Consensus Set of Quality Indicators
1. The MWM team identified existing indicators
a. Relevant to U.S. hospice and palliative care,
b. Available in the public domain as of October 2013, and
c. Developed through a rigorous process and/or tested for
reliability and validity.
2. The Technical Advisory Panel (TAP) rated indicators on their
scientific soundness and referred a set of indicators (n 34) for
review by the Clinical User Panel (CUP).
3. The CUP rated those indicators based on three dimensions:
a. How meaningful is this for patients/families?
b. How actionable is this for providers/organizations?
c. How large is the potential impact?
4. The CUP achieved consensus on the top 12 indicators for further
input.
5. The draft set of 12 indicators was posted on the American
Academy of Hospice and Palliative Medicine (AAHPM)s web site
to elicit feedback from AAHPM and Hospice and Palliative Nurses
Association (HPNA) members and their interdisciplinary teams,
and asked them to reduce the list of 12 measures down to 10 (or
less) of the best measures.
6. The draft set of 12 indicators was distributed to key relevant
external organizations and patient advocacy groups for a final set
of ratings.
7. The MWM team compiled the input from the membership,
teams, organizations, and advocacy groups for the final consensus
set of 10 indicators.
the Agency for Healthcare Research and Quality
(AHRQ) Quality Measures database, as well as indica-
tors included in the PEACE Palliative Care and Hos-
pice,8,12 Cancer Quality-ASSIST supportive
oncology,2,13 American Society for Clinical Oncology
(ASCO) Quality Oncology Practice Initiative
(QOPI) Care at the End of Life,10 and Assessing
Care of Vulnerable Elders (ACOVE) Palliative Care
and End of Life14,15 sets, indicators developed by the
National Hospice and Palliative Care Organization
(NHPCO)16,17 and American Medical Associatione
convened Physician Consortium for Performance
Improvement (PCPI),18 and indicators used by the
CMS. Indicators were considered relevant to hospice
and palliative care based on the numerator. If the
numerator or topic of the indicator was related to
the NCP domains or preferred practices of palliative
care, the indicator was considered for inclusion. Mem-
bers of the project steering committee then reviewed
the compiled list and identified any missing indica-
tors. From these sources, the MWM team identified
75 existing indicators that met criteria, and classified
them by NCP domain and compiled existing specifica-
tions and evidence about each indicator. Almost all in-
dicators had reliability and validity testing in palliative
care populations.
Panel Ratings and Public Comment
Technical Advisory Panel. The key purpose of the TAP
was to judge the technical strength of existing indica-
tors. The MWM team recruited 11 key researchers on
775
quality measurement in hospice and palliative medi-
cine for the TAP and co-chairs from AAHPM and
HPNA. After an introductory instructional webinar,
all members of the TAP were given detailed informa-
tion on the indicators and associated evidence,
including their reliability and validity testing, and
asked to review and rate the indicators on a scale
from one to nine, where one indicated low and nine
high scientific soundness (Appendix). When rating
each indicator, in addition to using the specifications
and evidence provided about each indicator, the
TAP members considered the AHRQ desirable attri-
butes of quality indicators:

Importance

Scientific soundness: clinical logic

Scientific soundness: indicator properties,
including evidence for reliability and validity

Feasibility
The TAP members rated each indicator twice: once
for the indicator as specified and once for the indica-
tor concept (the numerator only), and also could pro-
vide free-text comments. The purpose of the indicator
concept rating was to determine if the numerator was
a good candidate for future indicator development us-
ing a global all-palliative care population denominator
to be developed in the future.
The TAP members then participated in a confer-
ence call to discuss the indicators, focusing particu-
larly on those with disagreement, defined as at least
one rating lower than four or at least three ratings
lower than seven. Members could discuss issues raised
and advocate for or against indicators. The TAP mem-
bers then rated the indicators a second time using the
same process as the first round. This process identified
34 indicators sufficiently highly rated for scientific
soundness to advance to the CUP rating process.
Clinical User Panel. The purpose of the CUP process
was to select the indicators that were the most clini-
cally important and usable. The goal was to form a
concise measurement set of approximately 10 indica-
tors, although it was recognized that the final number
of indicators would be responsive to the concerns of
both comprehensiveness and brevity. The MWM
team recruited co-chairs from AAHPM and HPNA
and 27 members for the CUP to represent diversity
in settings where hospice and palliative services are
provided; patient populations and illnesses; and pro-
viders including physicians, nurses, social workers,
chaplains, and grief counselors, as well as consumer
groups and quality organizations. The CUP reviewed
the 34 indicators forwarded by the TAP using a similar
process as the TAP. The CUP rated the indicators on
meaningfulness for patients and families, actionability
for providers and organizations (including both
776
feasibility of measurement and possibility of making
changes based on results), and potential impact (prev-
alence of problem and/or amount of possible
improvement; Appendix). After the survey and discus-
sion webinar and further consolidation/prioritization
from the CUP co-chairs considering all feedback and
comments, consensus was reached on 12 cross-
cutting indicators from a variety of sources to submit
for the last steps in the process.
In the final steps, the MWM team distributed infor-
mation on the project and on the 12 indicators to the
AAHPM and HPNA membership and their teams and
to key relevant stakeholders for further ratings and
comment on which indicators were highest priority
and which should not be included, before winnowing
down to a final list of 10 indicators. In total, feedback
was received from 264 individuals, including 131 phy-
sicians, 106 nurses, 10 social workers, four pharmacists
and three chaplains, and 27 organizations, including
the American Academy of Neurology, American Acad-
emy of Pain Medicine, American Academy of Pediat-
rics, American College of Emergency Physicians,
American Geriatrics Society, Association of Profes-
sional Chaplains, Center to Advance Palliative Care,
NHPCO, National Palliative Care Research Center,
Oncology Nursing Society, Social Work Hospice &
Palliative Network, and The Joint Commission.
Results
Description of Selected Indicators
The final MWM quality indicator set includes 10 in-
dicators (Table 2). These include one in the NCP
domain of Structure and Process of Care (Compre-
hensive Assessment), three in the domain of Physical
Aspects of Care (Screening for Physical Symptoms,
Pain Treatment, and Dyspnea Screening and Manage-
ment), one in Psychological and Psychiatric Aspects of
Care (Discussion of Emotional or Psychological
Needs), one in Spiritual and Existential Aspects of
Care (Discussion of Spiritual/Religious Concerns),
and three in Ethical and Legal Aspects of Care (Docu-
mentation of Surrogate, Treatment Preferences, and
Care Consistency with Documented Care Prefer-
ences). In addition, the MWM list recommends that
a global indicator of care from the patient/family
perspective be included, but the MWM process did
not endorse a specific measure. Measures of the pa-
tient/caregiver experience that were considered and
met the initial MWM criteria for the candidate set
are described in Table 3. No indicators were included
in the domains of Social Aspects of Care, Cultural As-
pects of Care, and Care at the End of Life, high-
lighting gaps in areas where indicators may not be
Dy et al. Vol. 49 No. 4 April 2015
supported by current evidence or may potentially
not be appropriate for evaluating care.
None of the MWM indicators are disease-specific and
almost all apply to any patients receiving palliative or
hospice care services, although one is for vulnerable
elders (people older than 65 years at increased risk
for death or functional disability). Most are from the
PEACE data set and apply to hospitalized or hospice
patients; one (Discussion of Spiritual/Religious
Concerns) is for hospice only and one (Dyspnea
Screening and Treatment) is for ambulatory care (by
physicians). Most address processes of care. All indica-
tors would require medical record review or duplicate
data entry into a quality database, except for patient/
family perceptions of care. Only two are currently
endorsed by the NQF (Discussion of Spiritual/
Religious Concerns and Treatment Preferences).
Discussion
In summary, the MWM project, led by the AAHPM
and HPNA with input from a wide variety of stake-
holder and patient advocacy groups, assembled a
consensus-based concise set of process and outcome
quality indicators for internal current use for quality
measurement and improvement in settings caring
for hospice and palliative care patients. These indica-
tors cover five of the eight domains of the NCP Pallia-
tive Care Guidelines and most are applicable to broad
hospice and palliative care populations and more than
one setting (hospital and hospice care); only one is for
the ambulatory setting. Most are process indicators
and require medical record abstraction or separate
data entry. The MWM list also includes a global indica-
tor for patient/family experience of care, although no
consensus developed for a specific measurement tool
(the measure for hospices, the Family Evaluation of
Hospice Care, was being replaced at the time of this
project by the Consumer Assessment of Healthcare
Providers and Systems hospice survey) and the group
recommended that additional measure development
was needed.
Some of these indicators have been used and tested
more widely than others, and further refinement and
evaluation are needed. Many supporting tools already
exist for the PEACE indicators,12 and further develop-
ment of accompanying tools for the indicator set,
including details on standardizing data abstraction,
measurement specifications, benchmarking, and
evidence-based quality improvement, as well as poten-
tial recommendations for modifications of the indica-
tors based on experience with them, is needed. The
MWM panel recognized that quality indicators are
needed both to evaluate the quality of hospice and
palliative care programs and to evaluate the quality
Vol. 49 No. 4 April 2015 MWM in Hospice and Palliative Care 777

Table 2
Measuring What Matters Top 10 Quality Indicators Set
National Consensus
Project Domain Indicator Source

1. Structure and Comprehensive Assessment PEACE Set25,26

Processes of Care Hospice: Percentage of patients enrolled for more than seven days for whom a
comprehensive assessment was completed within five days of admission
(documentation of prognosis, functional assessment, screening for physical and
psychological symptoms, and assessment of social and spiritual concerns).
Seriously ill patients receiving specialty palliative care in an acute hospital setting: Percent of
all patients admitted for more than one day who had comprehensive assessment
(screening for physical symptoms and discussion of the patient/familys emotional
or psychological needs) completed within 24 hours of admission.
2. Physical Aspects Screening for Physical Symptoms PEACE Set25,26
of Care Percentage of seriously ill patients receiving specialty palliative care in an acute
hospital setting for more than one day or patients enrolled in hospice for more
than seven days who had a screening for physical symptoms (pain, dyspnea,
nausea, and constipation) during the admission visit.
Pain Treatment PEACE Set25,26
For seriously ill patients receiving specialty palliative care in an acute hospital setting
for more than one day or patients enrolled in hospice for more than seven days
who screened positive for moderate-to-severe pain on admission, the percent with
medication or non-medication treatment, within 24 hours of screening.
Dyspnea Screening and Management AMA-PCPI/NCQA18
Percentage of patients with advanced chronic or serious life-threatening illnesses
who are screened for dyspnea. For those who are diagnosed with moderate or
severe dyspnea, a documented plan of care to manage dyspnea exists (Ambulatory
physician care).
3. Psychological and Discussion of Emotional or Psychological Needs PEACE Set25,26
Psychiatric Aspects Percentage of seriously ill patients receiving specialty palliative care in an acute
of Care hospital setting for more than one day or patients enrolled in hospice for more
than seven days with chart documentation of a discussion of emotional or
psychological needs.
4. Social Aspects No indicators
of Care
5. Spiritual, Religious, Discussion of Spiritual/Religious Concerns Deyta, LLC/NQF# 1647
and Existential Percentage of hospice patients with documentation of a discussion of spiritual/
Aspects of Care religious concerns or documentation that the patient/caregiver/family did not
want to discuss.
6. Cultural Aspects No indicators
of Care
7. Care of the Patient at No indicators
the End of Life
8. Ethical and Legal Documentation of Surrogate PEACE Set25,26
Aspects of Care Percentage of seriously ill patients receiving specialty palliative care in an acute
hospital setting for more than one day or patients enrolled in hospice for more
than seven days with name and contact information for surrogate decision maker
in the chart or documentation that there is no surrogate.
Treatment Preferences PEACE Set/25,26NQF# 1641
Percentage of seriously ill patients receiving specialty palliative care in an acute
hospital setting for more than one day or patients enrolled in hospice for more
than seven days with chart documentation of preferences for life-sustaining
treatments.
Care Consistency with Documented Care Preferences ACOVE Palliative Care and
If a vulnerable elder has specific treatment preferences (for example, a DNR order, End of Life14,15
no tube feeding, or no hospital transfer) documented in a medical record, then
these treatment preferences should be followed.
Global Measure Although no specific global measure was endorsed by the MWM process, the
committee, panels, membership, and stakeholders agreed that patient and/or
family assessments of the quality of care are a key part of measuring quality for any
setting caring for palliative or hospice patients. Global measures that met the
MWM selection criteria are described in Table 3.
AMA-PCPI American Medical Association Physician Consortium on Performance Improvement; NCQA National Committee on Quality Assurance;
NQF National Quality Forum; DNR Do Not Resuscitate; ACOVE Assessing Care of Vulnerable Elders14,15 Palliative Care and End of Life.
More details on the measures are available at http://aahpm.org/quality/measuring-what-matters. The PEACE measures and supporting materials are available at
http://www.med.unc.edu/pcare/resources/PEACE-Quality-Measures.

of palliative care provided more broadly within set- these applications. Because the purpose of the project
tings by all providers to patients with palliative care was to define cross-cutting indicators, initiatives target-
needs, and further testing should evaluate both of ing quality measurement in specific populations (e.g.,
778 Dy et al. Vol. 49 No. 4 April 2015

Table 3
Candidate Global Measures of Patient and Family Experience Considered by the Measuring What Matters Project
Measure Description Source Comments

Family Evaluation of Hospice Care17
(FEHC; www.nhpco.org/fehc-
survey-materials)
Family Evaluation of Palliative Care
(FEPC; www.nhpco.org/fepc-
survey-materials)
Bereaved Family Survey (BFS;
www.cherp.research.va.gov/
PROMISE/)
Consumer Assessments and Reports
of End of Life27 (CARE; www.chcr.
brown.edu/pcoc/toolkit.htm)
NHPCO National Hospice and Palliative Care Organization; NQF National Quality Forum; CMS Centers for Medicare & Medicaid Services;
PROMISE Performance Reporting and Outcomes Measurement to Improve the Standard of care at End-of-life; VA Veterans Administration.
The FEHC survey is an after-death NHPCO/NQF# 0208 A new measure for CMS reporting
survey administered to bereaved is currently being developed
family caregivers of individuals
who died while enrolled in
hospice
A post-death survey that captures NHPCO The questions on the FEPC survey
family members perceptions are based on those in the FEHC
about the quality of the palliative survey, with wording
care that their loved ones modifications appropriate to
receiveddby a hospital-based palliative care service delivery.
consultant service or by a hospice
program offering palliative care
The purpose of this measure is to PROMISE Used throughout the VA; includes
assess families perceptions of the Center/NQF# 1623 questions specific to Veterans
quality of care that Veterans
received from the VA in the last
month of life. The BFS consists of
19 items (17 structured and two
open-ended).
The CARE survey is a mortality Center for Gerontology Requires administration by an
follow-back survey that is and Health Care interviewer.
administered to the bereaved Research, Brown
family members of adult persons University/NQF# 1632
(aged 18 years and older) who
died of a chronic progressive
illness receiving services for at
least 48 hours from a home
health agency, nursing homes,
hospice, or acute care hospital.

cancer) or settings (e.g., nursing homes) may benefit
from using more targeted indicators. In addition, the
purpose of the MWM project was not to identify or test
indicators for public reporting, which requires a
different evaluation process; the NQF, which evaluates
indicators for public reporting, has endorsed a num-
ber of indicators relevant to palliative and hospice
care (only two of the MWM set are endorsed by the
NQF).11 Finally, the MWM team emphasizes that
quality measurement is only one small component
of ensuring the best hospice and palliative
caredattention to structure, program development,
evidence-based practice, education, and quality
improvement are all also key to improving care for
patients and families.
The MWM list includes cross-cutting indicators in
five of the eight NCP domains, leaving social aspects
of care, cultural aspects of care, and care of the immi-
nently dying without any defined indicators in this set.
Although a number of indicators in these domains
were considered by the expert panels, the lack of
included indicators in these domains in the final
MWM set reflects the lack of scientific evidence and
insufficient documentation in the medical record
for feasible abstraction. In addition, some of these as-
pects, such as cultural aspects of care, are less prac-
tical for process and outcome quality measurement,
which requires issues that can be objectively
measured in standard ways that reflect quality across
populations. Potentially, structural indicators, such
as the availability or penetration of chaplaincy ser-
vices, may be more appropriate for these domains.
Some of the domains, such as physical and ethical
and legal aspects of care (which have the most indica-
tors), are more likely to be documented and easier to
objectively measure than others, such as social aspects
of care. For example, whether or not a patient has life-
sustaining treatment preferences documented in the
chart is easier to objectively measure than whether or
not the provider developed a comprehensive social
care plan, addressing issues such as relationships
and caregiver stress, as recommended by the NCP
guidelines.6
The lack of included indicators in the domain of
care of the imminently dying is a clear gap critical to
hospice and palliative care, where more objective
markers exist but further development is needed.
Eight of the nine indicators considered in this domain
were outcome measures of health care utilization (the
ninth was family evaluation of bereavement services;
the global indicators discussed previously also address
this domain from the patient/caregiver perspective).
In particular, two NQF-endorsed Quality Oncology
Practice Initiative indicators for cancer patients,
Vol. 49 No. 4 April 2015 MWM in Hospice and Palliative Care
whether or not a patient received hospice for less than
three days10,19 and proportion with more than one
emergency room visit in the last 30 days of life,19
were considered throughout the MWM process. How-
ever, many concerns about these two Quality Oncology
Practice Initiative indicators were raised by the panels,
membership, and stakeholder groups. These mea-
sures were tested using Medicare claims data and
have been benchmarked only for cancer patients;
and they require knowing about patients outcomes
outside a health system and who were within the last
30 days of life, which is generally challenging data to
obtain for palliative care providers not in hospice set-
tings. Outcome indicators also are generally risk-
adjusted and have exclusions, which have not yet
been developed for these indicators and could signif-
icantly affect outcomes. They also could be very
dependent on local structure, such as availability of ur-
gent care and the efficiency of hospice admission pro-
cesses. Finally, these indicators may not accurately
reflect patient and family perceptions of quality of
care, especially across cultures, as studies have found
that most caregivers feel that the length of stay in hos-
pice was sufficient20 and that caregivers in Japan did
not rank emergency department use highly as an indi-
cator of quality of care.21
Ideal indicators for care at the end of life could
include patient and family preferences for hospice
care and the timing of initial hospice referral and
admission accounting for whether any lag between
referral and admission was patient preference or a sys-
tem performance issue. On the other hand, outcomes
such as hospice length of stay can be integrative and
identify potential issues within a setting that need to
be addressed (e.g., very low hospice stays could indi-
cate that information is not being provided early
enough, referral processes are too slow, or coordina-
tion with hospices is poor). However, these would
need further detailed evaluation to identify the source
of the issue before attempts to improve quality can be
made.
The MWM process and the indicator set have a
number of limitations. The MWM set is based on an
extensive process of expert consensus and diverse
input, but will require ongoing efforts to further eval-
uate feasibility for routine data collection, usability
for quality improvement projects, links between pro-
cess and outcome indicators, and updates as new
and revised indicators become available. Using indica-
tors from the set will require careful monitoring for
unintended negative consequences, such as focusing
initiatives only on such areas where indicators are
robust and neglecting important areas where perfor-
mance may be more difficult to measure. Not all indi-
cators may apply to all settings or populations, and
779
programs will need to select those that are most
appropriate to their internal population, needs, and
opportunities for improvement. Further research
with benchmarking and evaluation of the impact of
contextual factors, such as electronic health record
system, academic status, or quality program structure,
are needed to improve the usability of the indicators.
None of the indicators have been evaluated in pediat-
rics, and evaluation specific to geriatric populations is
limited.
Patient/caregiver evaluations of care are critical to
hospice and palliative care, as medical record review
process indicators address only a small fraction of care
and cannot measure many important issuesdincluding
the quality of communication, coordination, respect,
and attention to patient preferences. However, there
is no universally agreed-on measure in palliative care;
different measures have been developed and are used
in different settings, and no current indicators we iden-
tified that met eligibility criteria crossed settings. The
consensus and stakeholder input processes, therefore,
did not endorse any specific measure, and further mea-
sure development is needed, but some process for
assessing relevant patient/caregiver input should be
included in settings providing care to palliative or
hospice patients.
Conclusion
In conclusion, the MWM Project of the AAHPM and
HPNA has assembled a list of 10 consensus indicators
for internal measurement of quality in settings serving
hospice and palliative patients. Although this set
serves as an initial starting place for standardizing
U.S. measurement and benchmarks for care, more
development and partnerships with organizations to
develop benchmarking is needed, and additional or
revised indicators may be needed for particular set-
tings or populations. Further research is needed to
evaluate how best to define palliative care populations
of interest for quality measurement, and how to mea-
sure the quality of hospice and palliative care both
overall in different settings and for patients served
by hospice and palliative care services. Finally, as eval-
uation of this national U.S. indicator set occurs,
consideration of including structural and newly devel-
oped indicators and coordination with lessons learned
from individual hospice and palliative care programs
in the U.S., relevant quality programs such as the
ASCO QOPI10 and other emerging international
palliative care quality measurement projects, such as
those in Australia,22 Belgium,23 and The
Netherlands,24 will help to maximize the value of qual-
ity measurement in the U.S.
780
Disclosures and Acknowledgments
This project was funded by the American Academy
of Hospice and Palliative Medicine, with additional
support from the Hospice and Palliative Nurses
Association.
The authors acknowledge the members of the Tech-
nical Advisory Panel: Marie Bakitas, Teresa Craig, Mary
Ersek, Chris Feudtner, Laura Hanson, Arif Kamal, Lisa
Lindley, Karl Lorenz, Carol Spence, Martha Tecca, and
Joan Teno; and Clinical User Panel: Michael Balboni,
Patricia Berry, Cynthia Boyd, Caitlin Brennan, Janet
Bull, Ira Byock, Barb Daley, Kenneth Doka, Jennifer
Eurek, Joy Goebel, Elizabeth Gunderson, Krista Harri-
son, Joan Harrold, Jean Kutner, Thomas Lee, Suzana
Makowski, Kelly McCutcheon Adams, Deirdre Mylod,
Marsha Nelson, Ritika Oberoi-Jassal, Lynn Reinke,
Michael Reynolds, Christine Ritchie, Eugenia Smither,
Lisa Stephens, Rodney Tucker, Deborah Waldrop, and
Joanne Wolfe. The authors also acknowledge June
Lunney and Steve Smith for their support and
guidance.
References
1. Institute of Medicine. Dying in America: Improving qual-
ity and honoring individual preferences near the end of life.
Washington, DC: The National Academies Press, 2014.
2. Walling AM, Tisnado D, Asch SM, et al. The quality of
supportive cancer care in the veterans affairs health system
and targets for improvement. JAMA Intern Med 2013;173:
2071e2079.
3. Fisch MJ, Lee JW, Weiss M, et al. Prospective, observa-
tional study of pain and analgesic prescribing in medical
oncology outpatients with breast, colorectal, lung, or pros-
tate cancer. J Clin Oncol 2012;30:1980e1988.
4. Teno JM, Clarridge BR, Casey V, et al. Family perspec-
tives on end-of-life care at the last place of care. JAMA
2004;291:88e93.
5. Dy S, Aslakson R, Wilson R, et al. Improving health care
and palliative care for advanced and serious illness. Closing
the quality gap: revisiting the state of the science. Evidence
Report No. 208. (Prepared by the Johns Hopkins University
Evidence-based Practice Center under Contract No. 290-
2007-10061-I.) AHRQ Publication No. 12(13)-E014-EF. Rock-
ville, MD: Agency for Healthcare Research and Quality,
2012.
6. National Consensus Project for Quality Palliative Care. Clin-
ical Practice Guidelines for Quality Palliative Care, 3rd ed.
2013; Available at: http://www.nationalconsensusproject.org.
Accessed September 1, 2014.
7. Center for Medicare & Medicaid Services. Hospice qual-
ity reporting. Available at: http://www.cms.gov/Medicare/
Quality-Initiatives-Patient-Assessment-Instruments/Hospice-
Quality-Reporting/index.html. Accessed September 23, 2014.
8. Hanson LC, Scheunemann LP, Zimmerman S,
Rokoske FS, Schenck AP. The PEACE project review of clin-
ical instruments for hospice and palliative care. J Palliat Med
2010;13:1253e1260.
Dy et al. Vol. 49 No. 4 April 2015
9. Nelson JE, Mulkerin CM, Adams LL, Pronovost PJ.
Improving comfort and communication in the ICU: a prac-
tical new tool for palliative care performance measurement
and feedback. Qual Saf Health Care 2006;15:264e271.
10. Campion FX, Larson LR, Kadlubek PJ, Earle CC,
Neuss MN. Advancing performance measurement in
oncology: quality oncology practice initiative participation
and quality outcomes. J Oncol Pract 2011;7:31se35s.
11. National Quality Forum. Palliative and end-of-life care
performance measures. Available at: www.qualityforum.org.
Accessed September 1, 2014.
12. The Carolinas Center for Medical Excellence (CCME).
PEACE Hospice and Palliative Care quality measures. Avail-
able at: http://www.med.unc.edu/pcare/resources/PEACE-
Quality-Measures. Accessed May 23, 2014.
13. Dy SM, Lorenz KA, ONeill SM, et al. Cancer quality-
ASSIST supportive oncology quality indicator set: feasibility,
reliability, and validity testing. Cancer 2010;116:3267e3275.
14. Lorenz KA, Rosenfeld K, Wenger N. Quality indicators
for palliative and end-of-life care in vulnerable elders.
J Am Geriatr Soc 2007;55:S318eS326.
15. Walling AM, Asch SM, Lorenz KA, et al. The quality of
care provided to hospitalized patients at the end of life.
Arch Intern Med 2010;170:1057e1063.
16. Connor SR, Tecca M, LundPerson J, Teno J. Measuring
hospice care: the National Hospice and Palliative Care Orga-
nization national hospice data set. J Pain Symptom Manage
2004;28:316e328.
17. Connor SR, Teno J, Spence C, Smith N. Family Evalua-
tion of Hospice Care: results from voluntary submission of
data via website. J Pain Symptom Manage 2005;30:9e17.
18. Wenger N, Dy SM, Casarett D, et al. American Medical
Association. Physician Consortium on Performance Improve-
ment palliative and end of life care physician performance
measurement set 2008; Available at: http://www.ama-
assn.org/ama/pub/physician-resources/physician-consortium-
performance-improvement/pcpi-measures.page. Accessed
September 4, 2014.
19. Earle CC, Park ER, Lai B, et al. Identifying potential in-
dicators of the quality of end-of-life cancer care from admin-
istrative data. J Clin Oncol 2003;21:1133e1138.
20. Teno JM, Shu JE, Casarett D, Spence C, Rhodes R,
Connor S. Timing of referral to hospice and quality of
care: length of stay and bereaved family members percep-
tions of the timing of hospice referral. J Pain Symptom
Manage 2007;34:120e125.
21. Miyashita M, Morita T, Ichikawa T, et al. Quality indica-
tors of end-of-life cancer care from the bereaved family
members perspective in Japan. J Pain Symptom Manage
2009;37:1019e1026.
22. Currow DC, Allingham S, Yates P, et al. Improving na-
tional hospice/palliative care service symptom outcomes sys-
tematically through point-of-care data collection, structured
feedback and benchmarking. Support Care Cancer 2015;23:
307e315.
23. Leemans K, Deliens L, Francke AL, et al. Quality indica-
tors for palliative care services: mixed-method study testing
for face validity, feasibility, discriminative power and useful-
ness. Palliat Med 2015;29:71e82.
Vol. 49 No. 4 April 2015 MWM in Hospice and Palliative Care 781

24. Claessen SJ, Francke AL, Sixma HJ, de Veer AJ,
Deliens L. Measuring relatives perspectives on the
quality of palliative care: the Consumer Quality Index Palli-
ative Care. J Pain Symptom Manage 2013;45:875e884.
25. Hanson LC, Rowe C, Wessell K, et al. Measuring pallia-
tive care quality for seriously ill hospitalized patients.
J Palliat Med 2012;15:798e804.
26. Schenck AP, Rokoske FS, Durham D, Cagle JG, Hanson LC.
Quality measures for hospice and palliative care: piloting the
PEACE measures. J Palliat Med 2014;17:769e775.
27. Teno JM, Clarridge B, Casey V, Edgman-Levitan S,
Fowler J. Validation of Toolkit After-death bereaved family
member Interview. J Pain Symptom Manage 2001;22:
752e758.

Appendix

Example Questions From the TAP and CUP Surveys