Audit for Registrars

Dr. Ramesh Mehay

Course Organiser
Bradford VTS
NOTE : Key points = core points to note for any
sytematic approach to audit
Definition
Clinical audit is the systematic and critical
analysis of the quality of clinical care.
This includes the procedures used for
diagnosis and treatment, the associated
use of resources and the effect of care on
the outcome and quality of life for the
patient.
Clinical Governance = improving
standards
Crombie et al. defined
Audit as the process of reviewing the
delivery of health care to identify
deficiencies so that they may be remedied.
Marinker (1990)
the attempt to improve quality of medical
care by measuring the performance in
relation to desired standards and by
improving on this performance


Definition less formal
Taking note of what we do
Learning from it
Changing it if necessary
With the aim of improving care

Why do It?
Development of professional education and self regulation
Improvement of quality of patient care
Increasing accountability
Improvement of motivation and teamwork
Aiding in the assessment of needs
As a stimulus to research
Clinical audit aims to lead to an improvement in the quality of service
providing:-
improved care of patients
enhanced professionalism of staff
efficient use of resources
aid to continuing education
aid to administration
accountability to those outside the profession


Fundamental Principles
All about improving patient care
Should be seen as part of day to day
practice
Developing a critical eye on what we are
doing
Trying to improve things all the time
The Audit Cycle
What Should Be Happening


What Is Happening?


What changes are needed
The Audit Cycle

What Audit Is Not
Not about:
Performance Appraisal of Staff
Disciplinary Actions
Needs Assessment
Research (which is usually about establishing
new knowledge)
Computers and Statistics
Competition between doctors
Never judge good and bad professionals based
on audit it is about improving care
Audit vs Research
Research Audit
Defines Good Practice Assess extent to which good practice
is being obeyed/improvements
May involve allocating pts to random
treatment groups
Never involves this
May involve placebo Rx Never involves placebo Rx
Disturbs the pt beyond that required
for normal clinical management
Never does this
May involvea completely new
treatment
Never involves a completely new
treatment
One off process Ongoing
Other notes
Both audit and Research are concerned with clinical practice effectiveness
Audit can contribute to research issues that need further exploration
When to Use What
Method When to use it Why
Research Good practice is not
defined and
comparisons are
needed
To define good
practice
Data Collection or
structured
observation
Practice patterns
unknown
To catalogue
prevailing practice
without making
judgements
Audit Good practice is
defined but we want
to know how much
we are sticking to it
To improve current
performance
Does Audit Lead to Change
Hearnshaw et al, BJGP 1998
Of 1257 audits
Around 80% on clinical care
Around 65% led to change
Making Audit Easier Avoid the
Blocks
BEFORE YOU START
Time big audits can eat up time in an already busy schedule, so :
Keep it simple and small
Look at one or two criteria
Engage the whole team otherwise it will be difficult! Is the team
ready? (Enthusiasm, wanting to improve)
WHEN YOU START
Delegate & Share the workload involve others
Make life easier use computers to do the laborious stuff (patient
searches)
Use protocols / standards already laid by others (why re-invent the
wheel?)
Be careful of data collection choose a topic which does not entail
too much data collection to the extent it becomes exhaustive with
subsequent loss of enthusiasm

Some Ideas
You can do an Audit of
Structure ie facilities being provided
Eg waiting times, availability of staff, record keeping (all patient
records should have a summary card), equipment
Process ie what was done to the patient eg referrals, prescribing,
investigations
Aspirin post MI, BP measurements 5 yearly in those aged 20-65
Outcome ie result for the patient
Eg patient satisfaction, patients with high BP aged between 20-35
should have a diastolic below 90mmHg within the first year of
treatment
high risk practices (significant event audits) eg pneumococcal
vaccines in splenectomised patients, are significant events being
acted upon?
The outcome is the ideal indicator for care but the most difficult to
measure.


Choosing a Topic
Condition has an important impact on health or of great local
concern KEY POINT ie serious consequences otherwise
Condition affects a large number of people
Good reasons for believing current performance can be improved or
improvements are needed KEY POINT
Convincing evidence about appropriate care is available
Data collection choose a topic which does not entail too much data
collection to the extent it becomes exhaustive with subsequent loss
of enthusiasm (? A pilot??)
CHOOSE SOMETHING THAT REALLY INTERESTS YOU
NO POINT AUDITING SOMETHING YOU THINK THE PRACTICE
IS DOING REALLY WELL
Then discuss with others are they interested too?


Choosing a Topic
Remember, topic should be important :
Chronic Disease Management eg referrals
or use of lab services (INRs in warfarin)
Preventative Care eg childhood imms,
Cervical Cytology
Prescribing eg aspirin post MI, PPIs (cost
issue)

Examples
Ways of spotting audit topics examples
Important clinical events admissions for asthma
Significant events patient died of MI no
record of smoking history or BP
Patients' complaints too long to get an
appointment
Observation no system for ensuring bag
drugs up to date
Observations of staff patient on Warfarin not had
INR for 6 months
NICE subjects post-MI patients on aspirin
Criteria
= yardsticks
An audit criterion is a specific statement of what
should be happening.
A statement which
A) defines a measurable item of health care
which
B) can be used to assess quality
KEYPOINT
Criteria should be explicit. You must
demonstrate evidence for justifying them
(literature search, Evidence Based!).
Criteria KEY POINTS
Ensure that the criterion is measurable
asthmatics should have had yearly PFs is
difficult to measure (how many years will you go back?);
asthmatics should have had a PF recorded in
the past year is more practical.
Dont try to audit too many criteria at once one or two
will keep you busy enough.
Try filling in the gaps of the following phrase to set your
audit criterion:
All patients with xxxxx should have had a xxxxx in the
last xxxxx.
Criteria
"All eligible women aged 25-65 should
have had a cervical smear in the last 5
years."
All asthmatics should have had a Peak
Flow recorded in the past year.
All drugs in our doctors bags should be
in-date.
Standards
An audit standard is a minimum level of
acceptable performance for that criterion.
Make sure the standard is directly related
to the criterion, also :-
Should include a suitable timeframe

Standards
Examples:
"At least 80% of eligible women aged 25-65 should have
had a cervical smear in the last 5 years."
At least 60% of asthmatics should have had a Peak
Flow recorded in the past year.
100% of drugs in our doctors bags should be in-date.

The standard should reflect the clinical and medico-legal
importance of the criterion.
in the example above, 80% of women should have had a
cervical smear,
But of those who've had an abnormal smear, 100%
should have had action taken.

Standards
How to set standards
Look at national guidelines
Literature (journals), textbooks
Local guidelines
Discussion with consultants/GPSIs
Discussion with trainer/partners
KEY POINT : Standards set should be realistic
and attainable. Justifiable reasons for the
standard set should be made explicitly clear.

Standards

Some criteria are so important that they need 100%
standard.
However, 100% standards are unusual patients or
circumstances usually conspire against perfection and
the standard needs to reflect that.
Your literature search should give you an idea of what
standards others have managed to reach.
Your standard needs to follow on directly from your
criterion for example,
Patients on thyroxine should have had TFTs done in the
last year; this should have happened in at least 90% of
patients.
TYING IT ALL TOGETHER
Examples of Standards & Criteria
Criteria Standards
All children under 2 years should
be immunised against tetanus and
polio
90% of registered patients under
the age of 2 years should have
been immunised against polio and
tetanus
All notes of those patients with an
allergy to penicillin should be
marked
95% of patients with an allergy to
penicillin should be clearly marked
All patients in the surgery should
wait no longer than 30 minutes
before a consultation
70% of patients in the surgery
should wait no longer than 30
minutes before a consultation
Preparation & Planning
Must show evidence of teamwork
otherwise you will fail
Data Collection (1)
You can collect information from:
computer registers
review of contents of medical records
questionnaires patients, staff or GPs
data collection sheets


Data Collection
Be careful of data collection choose a topic
which does not entail too much data collection to
the extent it becomes exhaustive with
subsequent loss of enthusiasm
? Sampling random or systemic
Only collect essential information
Use computers, ?data collection forms
Use other staff & delegate dont do all the work
yourself
Set a deadline

Presenting the Results
Collect Results
Analyse Results
Summarise Results
Present Results to the team
Simple arithmetic calculations
Use percentages
Results of 2
nd
data collection presented in
the same way as the 1
st

Discussion Data Collection 1

Comparing Results to Standards
Criterion Standard Observed Result
All patients should be
seen within 15
minutes of their
appointment
timeMinimum
70% 45%
70%45%All diabetics
to have had HbA1C in
last 3
monthsMinimum
95% 90%
90%Drug allergies to
be marked as active
problem on
computerMinimum
100% 95%
Discussion Data Collection (1)
KEY POINT (Discussion of Data Collection
1) : You need to explain why you think the
practice didn't meet the standard that was
set.


Discussion why standards not
met
Think: What reasons are there for practices not meeting audit
standards?

For example : reasons have included:
Practice reasons:
Results having been put down as free text on computer,
rather than coded;
Opportunistic rather than formal recall system in use;
Doctor reasons:
Not all GPs were aware of the practice policy;
Not all partners agreed with the policy;
Patient reasons:
Patients refusing to have tests done;
Patients on holiday when tests due.
Implementing Changes
The most challenging stage
Audit can tell you whether changes are
needed, but it cant tell you what methods
to use

Implementing Change
The changes to be implemented should be
a team discussion and decision (?a
practice meeting)
What to do at the Practice Meeting:
Emphasise what has been achieved.
What are we proud of?
What are we not so proud of?
How can we correct any deficiencies?

Implementing Change
Changes must be practical!
How are you actually going to make the changes?
Simply saying Weve got to do better wont result in
change
You need to think through in detail
what needs to be done
whos going to do it
when
and how.
If you get very low results, you may consider resetting
the standards to a more realistic level (but justify it)

Implementing Change
KEYPOINT
Just telling people to do things better won't result in
change. You need to write up in some detail how the
changes will take place.
FAIL Example:
"The GPs agreed to do a serum rhubarb on any patient that they see
who is on Viagra" - fail - this wouldn't be likely to pass, as there is no
system to help them remember.
PASS Example :
(a) The GPs were given a prompt card that they could stick on their
computer screen as a reminder to do a serum rhubarb on any
patient that they see who is on Viagra;
(b) the secretary will search every three months for patients who are
overdue for their serum rhubarb, and flag it as an active problem on
the computer system" - pass - as it should result in change.
Closing the Loop
Ie repeating the cycle
Re-evaluate care to ensure that any
remedial action has been effective.
Audit is a continuous cycle if you didnt
meet the standard and youve planned
changes, youll need to repeat the audit to
make sure the changes have happened.
Conclusions from the Audit
Summary of main issues learned
KEYPOINTS:
Comment on any improvements that have resulted.
How well did your proposals for change work?
If you again didn't reach the standard that you set, why
not?
If you did, should you be aiming higher next time, or look
at something else e.g. whether abnormal serum
rhubarbs have actually been acted on?
Where should the practice go from here
Useful Resources
MAAGs medical audit advisory groups
Clinical Governance Advisory Groups
National/Local Guidelines
RCGP database of simple tested audits for
day-to-day use
Literature, Books
The WWW
Consultants, GPSIs, Trainers, Partners

How To Fail
No justification for choice of audit
No justification for criteria/standard settings
Not having explicit criteria/standards
Setting unreasonable standards
A general lack of evidence based literature or using material that is not peer
referenced
Not explicitly displaying teamwork in the method must give specific
examples
Numerical errors re: data collection
Presentation of data collection eg no graphs, no percentages (ie the reader
has to do the hardwork him/herself)
Not giving much thought to changes to be evaluated and not being specific
enough. Not delegating specific changes to specific people/persons.
Poor conclusions and what the process has taught you
No inclusion for possible sources of bias
References not properly quoted
IF YOU DONT WANT TO FAIL
Go through the following online tutorial
http://www.mharris.eurobell.co.uk
Look at the Marking Schedule (yes, they
provide you with an answer sheet!)
www.mharris.eurobell.co.uk/marking.htm
You must pass on all 8 criteria.
SHOs doing Audit for Summative
Assessment
If you are doing the audit while an SHO, you
need to choose a topic that looks at the GP-
hospital interface. Referrals or discharge letters
are possible areas for audit. Again, you need to
demonstrate that you've found a problem that
needs to be investigated.
I suggest that you discuss your proposed
audit with your GP Scheme Organiser before
you go ahead - your hospital colleagues may not
know what's needed for Summative
Assessment.
Checking GPR Understanding
DISCUSS THE FOLLOWING STATEMENTS
An example of the Audit of process is audit of referrals to hospitals.
Audit usually consumes an extensive amount of resources (of time, money
etc.).
Rare conditions should be audited.
The higher the standard the practitioner starts with, the stronger is the
resulting audit.
Maintaining clearly written notes of at least 20% of patients who are
sensitive to penicillin is an acceptable standard in general
practice.
The higher the amount of data the practitioner collects, the easier is the
decision making process in audit.
The most challenging stage in Audit is implementing change.
In data collection all in the target population must be included.
The agreed standards can be reset at realistic percentages after the first
round of data collection.

Clinical Audit Association Ltd
Clinical Audit Association Ltd
Cleethorpes Centre
Jackson Place
Wilton Road
Hunberton
Lincolnshire DN36 4AS
Tel: 01472 210 682
http://www.the-caa-ltd.demon.co.uk
Clinical Governance Research
and Development Unit

Dept of General Practice and Primary Health
Care
University of Leicester
Leicester General Hospital
Gwendolen Rd
Leicester LE5 4PW
Tel: 0116 258 4873
Fax: 0116 258 4982
email: cgrdu@le.ac.uk
http://www.le.ac.uk/cgrdu
Cochrane Database of
Systematic Review

020 7383 6185
c/o
British Medical Association
BMA House
Tavistock Square
London WC1H 9JP
NICE

11 Strand
London
WC2N 5HR

Tel: 020 7766 9191
Fax: 020 7766 9123

http://www.nice.org.uk
RCGP Effective Clinical Practice
Unit
School of Health and Related Research
Regent Court
30 Regent Street
Sheffield S1 4DA
Tel: 0114 222 5454
Fax: 0114 272 4095
Email: scharrlib@sheffield.ac.uk
http://www.shef.ac.uk/~scharr/
RCGP NE Scotland Faculty
The Primary Care Resource Centre
Foresterhill Road
Aberdeen AB25 2ZP
Tel: 01224 558 042
Fax: 01224 558 047
Email: rcgp@pcrc.grampian.scot.nhs.uk
http://www.rcgp.org.uk/rcgp/faculties/
nescot/index.asp
UK Cochrane Centre
Dr Iain Chalmers, Director
NHS Research and Development
Programme
Summertown Pavilion
Middle Way
Oxford OX2 7LG

Tel: 01865 516300
Cochrane Collaboration in the
field of primary care
For information concerning work by the
Cochrane Collaboration in the field of primary care,
contact:
Dr Lorne Becker
Professor and Chair
Dept of Family Medicine
SUNY Upstate Medical University
475 Irving Avenue
Syracuse, NY 13210
USA
Tel: +1 315 464 7010
Fax: +1 315 464 6982
E-mail: beckerla@hscsyr.edu
http://www.update-software.com/ccweb/
default.html
NHS Centre for Reviews and
Dissemination
University of York
York, YO1 5DD
Tel: 01904 433 634
Fax: 01904 433 661
Email: revdis@york.ac.uk
http://www.york.ac.uk/inst/crd
Sources
This power point has been derived from :
http://www.mharris.eurobell.co.uk
http://kims.org.kw/bulletin/Issues/issue2/A
udit.pdf