See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/282517661

Usability of Incident Reporting Systems: Preliminary Results of a Case Study

Article · June 2015

DOI: 10.1177/2327857915041031

CITATIONS READS
2 222

2 authors, including:

Esa Rantanen
Rochester Institute of Technology
110 PUBLICATIONS   664 CITATIONS   

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Course on I/O Psychology View project

All content following this page was uploaded by Esa Rantanen on 07 March 2016.

The user has requested enhancement of the downloaded file.

 2015 International Symposium on Human Factors and Ergonomics in Health Care: Improving the Outcomes 168

USABILITY OF INCIDENT REPORTING SYSTEMS:

PRELIMINARY RESULTS OF A CASE STUDY
Paul Wiele and Esa Rantanen
Department of Psychology, Rochester Institute of Technology, Rochester, NY

Although low participation in incident-reporting systems contributes to the problem of poor error preven-
tion, little research has focused on improving participation. The present study is a case study of a hospital’s
incident reporting practices, seeking to increase both the number and quality of their incident reports by ex-
amining the usability of the reporting system itself as well as organizational factors impacting attitudes to-
wards reporting. A snowball sample of hospital staff was recruited to identify organizational and software
usability factors that both impede and promote frequent and useful incident reporting. Preliminary results
suggest that time pressure is a key impediment to reporting, and that hospital staff in certain units are al-
ready attempting innovative shortcuts to cut time demands without reducing their reporting rates.

INTRODUCTION Is Incident Reporting Working?

Thanks in large part to the “Quality Chasm Series” of re-
ports (Institute of Medicine, 1999, 2001, 2004), recent efforts
to improve patient safety in the American healthcare system
have focused on adverse events, or cases of preventable acci-
dental harm to patients. Adverse events, along with near miss-
es, which could plausibly have caused harm but did not, to-
gether make up the general category of “incidents”. Figure 1
illustrates this taxonomy and reporting protocols in New York
State.
Near misses generally stem from the same underlying
problems and processes as actual incidents (Institute of Medi-
cine, 2004). They are also much more numerous than incidents
Copyright 2015 Human Factors and Ergonomics Society. All rights reserved. DOI 10.1177/2327857915041031
It is very difficult to judge the extent to which an incident
reporting system is working because the true number of inci-
dents is always unknown (Bleich, 2005). Reporting rates vary
in nonrandom ways that suggest that they are not representa-
tive of the true incident rates. Even in facilities with proce-
dures in place for making reports, they likely capture only a
small fraction of the incidents. For example, Jha et al. (1998)
assessed patient charts and incident reports for evidence of ad-
verse drug events (ADEs) and found that only 9 out of 166 in-
cidents classified as “preventable ADEs” had been voluntarily
reported.
Murff, Patel, Hripcsak, and Bates (2003) built on this and

that result in harm, allowing for more reliable detection of pat-
terns and trends through statistical means. Hence, in addition
to learning from adverse events, a system designed to mini-
mize future harm to patients would also do well to learn from
near misses.
Figure 1. A taxonomy of incidents in health care and reporting
protocols in New York State.
other studies to compare voluntary incident reporting with
other methods for detecting incidents such as chart review and
usability testing. They concluded that all methods have limita-
tions and no method alone can detect all incidents. Other
sources of variation in incident reporting include geographical
differences, including within states that require some form of
incident reporting (State of New York Department of Health,
2007), and differences by medical specialty (Busse & Wright,
2000; Johnson, 2002). A possible explanation for this is that
policy and safety culture differences on the level of individual
facilities or units contribute to the willingness of healthcare
workers to report incidents.
There has been some recent progress from the Partnership
for Patients initiative, which has been credited with accelerat-
ing the recent decline in hospital-acquired conditions (Agency
for Healthcare Research and Quality, 2013). Nevertheless, the
healthcare system in general still lacks formal ways of sharing
safety information that would allow safety improvements to
flow freely through the system.
The highly successful Aviation Safety Reporting System
(ASRS) offers an example of how such a system could be im-
plemented, and a starting point for the criteria that make a re-
porting system effective. Most importantly, the reporting sys-
tem should allow confidentiality to protect the reporter from
professional or legal risks as well as feedback that encourages
future reporting (Billings, 1998). ASRS also offers us the in-
triguing and unusual insight that minimizing harm may not re-
quire a complete or even representative record of incidents,

Downloaded from hcs.sagepub.com at HFES-Human Factors and Ergonomics Society on January 6, 2016
2015 International Symposium on Human Factors and Ergonomics in Health Care: Improving the Outcomes
but instead the ability for “sharp end” workers to identify and
report those issues they reasonably believe to be the most im-
portant (Beaubien & Baker, 2002).
Approaches to Safety
To accomplish the goal of identifying the most important
types of incidents, our research focused specifically on the ap-
proach of increasing participation in the incident reporting
process. This requires an understanding of those factors that
inhibit reporting and those that actively encourage it—in other
words, as Hollnagel, Wears, and Braithwaite (2015) would put
it, a “Safety-II” attitude of understanding why a system suc-
ceeds in addition to why it fails. These inhibitions and encour-
agements can come from the reporting system itself or from
the practices of the organization that implements it. The re-
search reported here examines both in the case study of Roch-
ester General Hospital (RGH), which uses a web-based appli-
cation to report incidents.
Employees’ willingness to report incidents within any or-
ganization is partially a product of that organization’s safety
culture, that is, its attitudes and practices regarding improving
safety. An organization with a weak safety culture will blame
individual employees for incidents, and consider reporting
those incidents “out of line”, while one with a strong safety
culture will credit employees for providing useful information
that can be learned from (Taylor, 2012). Safety culture indica-
tors show that better teamwork (Erler, et al., 2013) and strong
safety culture on the level of the whole organization (Kagan &
Barnow, 2013) predict a higher incident reporting rate and a
lower actual rate of incidents occurring.
Current Practices
Rochester General Hospital (RGH) uses multiple special-
ized methods to collect information about incidents. Billing
data may be used to measure factors like time of stay, which
can be influenced by the quality of care provided. The nursing
department submits data to the National Database of Nursing
Quality Indicators. Finally, hospital staff in general report in-
cidents using an incident reporting web application produced
by Quantros, Inc. (this app is commonly referred to by users as
“Quantros” for short). Because Quantros alone among these
methods identifies both adverse events and near misses, is
used by all departments, and contains details about not just
what kinds of incidents occur but also how they occur, this re-
search will focus on the usability and organizational practices
surrounding Quantros.
Purpose of the Study
The purpose of this research is to examine incident report-
ing in a large general hospital in a comprehensive and system-
atic way. This project has three distinct stages: (1) Examina-
tion of the user experiences with the incident reporting inter-
face through a focus group study, (2) investigation of the gen-
eral safety culture in the hospital through in-depth interviews
and surveying of hospital staff members, and (3) examination
of the analyses of the reported incident data and dissemination
Downloaded from hcs.sagepub.com at HFES-Human Factors and Ergonomics Society on January 6, 2016
169
of the analysis results. In this paper we report the results from
the first stage of the study.
METHODS
Participants
Participants were recruited via “snowball” sampling
among staff in any department or role at the hospital. Initial e-
mails were sent by the hospital’s Patient Safety department to
likely participants. Those who volunteered were then asked to
recruit coworkers who they thought would be interested in
participating.
Materials
Participating in the first stage of the study entailed a usa-
bility test and a structured interview. So far two usability test
and interview sessions have been conducted, with a goal of a
total of 10. Additional preliminary usability data were gath-
ered by the authors in a pilot test.
In the usability testing phase, participants were recorded
entering an incident report about a hypothetical scenario based
on a published case report (Gupta & Cook, 2013) using the
Quantros training module. This module duplicates the form
used to report incidents, without actually submitting the report
when complete. The recordings are reviewed so that any prob-
lems the participants encounter may be noted and classified
according to usability guidelines for web forms (Bargas-Avila
et al., 2010) as well as the more common and general- purpose
heuristic analysis (Nielsen, 1994).
Interviews
The interview phase began with general questions as-
sessing the participant’s views on the organizational culture.
The questions are based on those used in existing safety cul-
ture surveys (Commission on Accreditation of Medical
Transport Systems, 2012; Agency for Healthcare Research
and Quality, 2014b, 2014a), one study regarding the imple-
mentation of a health information exchange system (Feldman,
Schooley, & Bhavsar, 2014), and one study on eliciting the
user requirements of a proposed software project (Browne &
Rogich, 2001). The remainder of the interview phase consists
of two “walkthroughs” using the critical incident technique
(Crandall, Klein, & Hoffman, 2006). In the first critical inci-
dent walkthrough, the participant is asked to guide the inter-
viewer through an example of a time they decided not to make
an incident report, and in the second a time they decided to
make one, in both cases describing the circumstances and
thought processes that led to those outcomes.
PRELIMINARY RESULTS
Pilot testing of the application’s usability using the au-
thors as naive users detected potential usability problems in a
variety of different categories. Categories are described in Ta-
ble 1 for the web form guidelines from Bargas-Avila et al.
(2010) and in Table 2 for the general usability heuristics from
2015 International Symposium on Human Factors and Ergonomics in Health Care: Improving the Outcomes 170

Nielsen (1994). However, tests on experienced users have on-
ly corroborated the total length of the form and the large num-
ber of options on menus as being actual inhibitors to use.
Form length is in the “form content” category under the web
form-specific guidelines and the “visibility of system status”
category under the general heuristics. The large number of op-
tions is in the “input types” category under the web form
guidelines but is not accounted for by the general heuristics.
Further tests will provide further evidence for or against the
importance of other categories of usability issues.
Some problems encountered in pilot usability testing but
not corroborated so far in tests on experienced users remain
causes for concern. Responding to questions has inconsistent
results “downstream” in the form. The system often causes
additional fields to appear contingent on those answers, but
sometimes produces pop-ups with their own "save" and "can-
cel" buttons (Table 1, "form content"; Table 2, "consistency
and standards"). Some menus lack a way to indicate "not ap-
plicable" when such a response would be expected, such as
when answering the question "was another area involved?"
when describing where the incident took place (Table 1, "input
types"). The software provides no feedback to the user to con-
firm when a report in progress has been saved (Table 2, "visi-
bility of system status"). Perhaps most alarmingly, other than a
tutorial video, there is no help documentation, so troubleshoot-
ing requires contacting a support center in ways that would
not guarantee anonymity (Table 2, "help and documentation").

Table 1
Reporting Application’s Compliance with Bargas-Avila and Colleagues’ (2010) Guidelines for Web Forms

Category Description Violations: Pilot Main

Form content Forms should be short and intuitively-ordered, allow content flexibility in answers, and clearly Yes Yes
distinguish between required and optional questions.
Form layout Fields for answers should be labeled above, listed one layout per row, and sized appropriately No No
for the length of the answer.
Input types Menus, checkboxes, buttons, or open response fields are appropriate for different numbers and Yes Yes
types of answers. Options should be in an intuitive order and limited in number if possible.
Error handling Answers should have their expected formats clearly indicated and should not be cleared by er- No No
rors. Error messages should be polite, informative, embedded in the form, and easily noticed.
Form submission Submission buttons should only be usable once and - not be confusable with “reset” buttons. Yes No
Submissions and how they will be used should be confirmed.

Table 2:
Reporting Application’s Compliance with Nielsen’s (1994) General Usability Heuristics.

Heuristic Description Violations: Pilot Main

Visibility of system status The system should provide appropriate and timely feedback Yes Yes
Match between system and the The system should present things in familiar terms similar to those encoun- Yes No
real world tered in other contexts
User control and freedom The system should allow users to undo and redo actions Yes No
Consistency and standards Terms should used consistent and conventional definitions Yes No
Error prevention The system should be designed to avoid errors rather than simply address Yes No
them
Recognition rather than recall The user should have easy access to references and not need to store and Possibly No
recall information from one part to another
Flexibility and efficiency of use There should be customization and shortcut tools available for experienced Possibly No
users to adapt the system to their needs
Aesthetic and minimalist design Dialogues should avoid displaying unnecessary information No No
Help users recognize, diagnose, Error messages should be presented in terms the user can understand and No No
and recover from errors suggest ways to address the error
Help and documentation Help tools should be readily available, searchable, and explain things step by Yes No
step

The structured interviews revealed generally positive
opinions of the hospital’s safety culture, although with the ca-
veat that participants believe other, unspecified units to be less
safety-conscious and less dedicated to learning from incidents
than their own. Participants reported that the amount and qual-
ity of feedback provided to staff varies between units. Some
units were seen as much more punitive than others, although
there is an official effort in the form of an in-hospital newslet-
ter to remind staff of the usefulness of incident reporting in
general and near miss reporting in particular. Coordination be-

Downloaded from hcs.sagepub.com at HFES-Human Factors and Ergonomics Society on January 6, 2016
2015 International Symposium on Human Factors and Ergonomics in Health Care: Improving the Outcomes
tween units when major policy changes need to be made is dif-
ficult, and there does not seem to be a procedure in place to
collect and compare data from different units to look for simi-
lar incidents or solutions to them elsewhere in the hospital.
Both the emergency department and a specialized cardi-
ology unit reported suffering serious time pressure problems,
with staff feeling that they did not have sufficient time to
make incident reports without being interrupted. As a result,
both units developed shortcuts to try to make incident report-
ing faster without sacrificing quality. In the ED, with the hos-
pital’s official support, a telephone hotline has been intro-
duced, while in the cardiology unit, a system involving short
Figure 2. The flow of information within the incident reporting system. Dashed lines indicate problematic areas.
DISCUSSION
The above descriptions, in addition to some preliminary
information provided by the Patient Safety department before
data collection began, allow us to describe the methods in
place for reporting incidents (Fig. 2). An incident may be re-
ported directly into the web app or via one of the unit shortcut
methods. Designated quality coordinators receive reports and
ensure that the unit in which the incident took place is notified
to follow up on it. Unit managers then decide how to respond
to the incidents in their department, and request additional as-
sistance from upper management when needed. Although par-
ticipants reported that this is sometimes extremely helpful—it
was through this process that the emergency hotline shortcut
became an official hospital policy—it also falls short in other
Downloaded from hcs.sagepub.com at HFES-Human Factors and Ergonomics Society on January 6, 2016
171
written narratives accompanied by stickers containing patient
information fills the same niche. In both cases, the narratives
are submitted to designated staff members, who transcribe and
classify them in the reporting application. Participants esti-
mated that this cut the time for the typical staff member to
submit a report from 15–20 to 1–2 minutes, while also pre-
serving enough information for unit managers and other ana-
lysts to make minor local policy changes. Information is not
available at this time to judge the effectiveness of these local
changes. Figure 2 describes the information flow within the
incident reporting system in place at RGH, based on our re-
search on the issues.
ways by not facilitating sharing of data between units. It is al-
so worth noting the other information flow represented by a
dotted line, from the incident reporting newsletter. The news-
letter provides a highly condensed form of feedback, and we
have not yet been able to evaluate whether this level and for-
mat of feedback is encouraging.
Usability heuristic assessment and comparison to web
form guidelines point to several likely factors discouraging
system use: time constraints due to the amount of information
required by Quantros, lack of customizability and shortcuts for
experienced users, and inconvenient error handling/technical
support. Comparison of known organizationally-influenced
features of how Quantros is used to those of a known success-
ful reporting system (ASRS) suggests additional likely dis-
couraging factors: lack of evidence that incident reports will
2015 International Symposium on Human Factors and Ergonomics in Health Care: Improving the Outcomes
be used to improve patient safety, fear of punishment, and lack
of sufficient organizational encouragement to continue partic-
ipating. Additionally, common work experience suggests that
the issue of lack of sufficient time to provide a report could al-
so be due to time not being available in the potential user’s
work schedule.
Further Research
At this point, we have not gathered sufficient data to de-
termine trends that contribute to more or less, or better or
worse, reporting. Additional interviews and usability tests are
needed, and the participant pool must include both unit man-
agers, to determine whether they are lacking important details
needed to understand the causes of incidents, and ordinary
staff members, to determine whether the software and the or-
ganization are truly providing the support they need to report
everything that is worth reporting. These will provide the
“depth” of the study, identifying more notably bad and notably
good factors affecting incident report frequency and useful-
ness. The “breadth” will come in a final stage of data collec-
tion, a hospital-wide survey gauging the staff’s experience
with these different factors.
The ultimate goal of this study is to promote the use of in-
cident reporting in such a way that “sharp end” staff feel not
only able but actively encouraged to identify and fix the most
important safety problems they encounter. To this end, we will
use the survey results to categorize issues by their common-
ness and severity, which we can then use to recommend spe-
cific improvements based on a review of the literature to iden-
tify where others have already succeeded with similar prob-
lems. It is our hope and goal that this will encourage im-
provement both locally and at other facilities in the “Safety-I”
sense of learning from past incidents and in the “Safety-II”
sense of noting where and why successes occur.
REFERENCES
Agency for Healthcare Research and Quality. (2013). Interim
update on 2013 annual hospital-acquired condition rate
and estimates of cost savings and deaths averted from
2010 to 2013 (Report No. 15-0011-EF). Retrieved from
http://www.ahrq.gov/professionals/quality-patient-safety/
pfp/interimhacrate2013.pdf
Agency for Healthcare Research and Quality. (2014a). Hospi-
tal survey on patient safety culture: Items and dimensions.
[Supplemental materials for data collection instrument].
Rockville, MD. Retrieved from http://www.ahrq.gov/ pro-
fessionals/quality-patient-safety/patientsafetyculture/ hos-
pital/index.html
Agency for Healthcare Research and Quality. (2014b). Hospi-
tal survey on patient safety. [Data collection instrument].
Rockville, MD. Retrieved from http://www.ahrq.gov/ pro-
fessionals/quality-patient-safety/patientsafetyculture/ in-
dex.html
Bargas-Avila, J., Brenzikofer, O., Roth, S., Tuch, A., Orsini,
S., & Opwis, K. (2010). Simple but crucial user interfaces
in the world wide web: Introducing 20 guidelines for usa-
Downloaded from hcs.sagepub.com at HFES-Human Factors and Ergonomics Society on January 6, 2016
172
ble web form design. In R. Mátrai (Ed.), User interfaces
(pp. 1–10). Rijeka, Croatia: InTech.
Beaubien, J. M., & Baker, D. P. (2002). A review of selected
aviation human factors taxonomies, accident/incident re-
porting systems and data collection tools. International
Journal of Applied Aviation Studies, 2(2), 11–36.
Billings, C. (1998). Incident reporting systems in medicine
and experience with the aviation safety reporting system.
In R. Cook, D. Woods, & C. Miller (Eds.), A tale of two
stories: contrasting views of patient safety. Report from a
workshop on assembling the scientific basis for progress
on patient safety. McLean, VA: National Patient Safety
Foundation at the AMA.
Bleich, S. (2005). Medical errors: five years after the IOM re-
port. Issue Brief (Commonwealth Fund), 830, 1–15.
Browne, G., & Rogich, M. (2001). An empirical investigation
of user requirements elicitation: Comparing the effective-
ness of prompting techniques. Journal of Management In-
formation Systems, 17, 223–249.
Busse, D., & Wright, D. (2000). Classification and analysis of
incidents in complex medical environments. Topics in
health information management, 20(4), 1–11.
Commission on Accreditation of Medical Transport Systems.
(2012). Safety culture survey [survey instrument]. Re-
trieved from http://www.camts.org/Safety-Culture-
Survey.html
Crandall, B., Klein, G. A., & Hoffman, R. R. (2006). Working
minds: A practitioner’s guide to cognitive task analysis.
MIT Press.
Erler, C., Edwards, N. E., Ritchey, S., Pesut, D. J., Sands, L.,
& Wu, J. (2013). Perceived patient safety culture in a crit-
ical care transport program. Air Medical Journal, 32(4),
208–215.
Feldman, S., Schooley, B., & Bhavsar, G. (2014). Health in-
formation exchange implementation: Lessons learned and
critical success factors from a case study. Journal of Med-
ical Internet Research Medical Informatics, 2(2), e19.
Gupta, K., & Cook, T. (2013). Accidental hypoglycaemia
caused by an arterial drug flush error: A case report and
contributory causes analysis. Anaesthesia, 68, 1179–1187.
Hollnagel, E., Wears, R., & Braithwaite, J. (2015). From Safe-
ty-I to Safety-II: A white paper. Authors.
Institute of Medicine. (1999). To Err is Human: Building a
safer health system. Committee on Health Care in Ameri-
ca. Institute of Medicine. Washington, DC: National
Academy Press.
Institute of Medicine. (2001). Crossing the quality chasm: A
new health system for the 21st century. Washington, DC:
National Academies Press.
Institute of Medicine. (2004). Patient safety. Washington, DC:
National Academies Press.
Jha, A., Kuperman, G., Teich, J., Leape, L., Shea, B., Ritten-
berg, E., & Bates, D. W. (1998). Identifying adverse drug
events: Development of a computer-based monitor and
comparison with chart review and stimulated voluntary
report. Journal of the American Medical Informatics As-
sociation, 5(3), 305–314.
Johnson, C. (2002). Reasons for the failure of incident report-
ing in the healthcare and rail industries. In Components of
2015 International Symposium on Human Factors and Ergonomics in Health Care: Improving the Outcomes 173

system safety: Proceedings of the 10th safety-critical sys-
tems symposium (pp. 31–57). Springer.
Kagan, I., & Barnoy, S. (2013). Organizational safety culture
and medical error reporting by Israeli nurses. Journal of
Nursing Scholarship, 45(3), 273–280.
Murff, H., Patel, V., Hripcsak, G., & Bates, D. (2003). Detect-
ing adverse events for patient safety research: A review of
current methodologies. Journal of Biomedical Informat-
ics, 36, 131–143.
Nielsen, J. (1994). Heuristic evaluation. In J. Nielsen & R.
Mack (Eds.), Usability Inspection Methods (pp. 25–62).
New York, NY: John Wiley & Sons.
State of New York Department of Health. (2007). New York
Patient Occurrence Reporting and Tracking System—
NYPORTS—2005, 2006, 2007 Report (Tech. Rep.). Alba-
ny, NY: State of New York Department of Health.
Taylor, J. B. (2012). Safety culture: Assessing and changing
the behaviour of organisations. Gower Publishing, Ltd.

Downloaded from hcs.sagepub.com at HFES-Human Factors and Ergonomics Society on January 6, 2016
View publication stats