For the full versions of these articles see bmj.

com

PRACTICE

THE COMPETENT NOVICE

Audit: how to do it in practice

Andrea Benjamin

BMJ Learning, BMA House, London WC1H 9JR ABenjamin@bmjgroup.com

BMJ 2008;336:1241-5 doi:10.1136/bmj.39527.628322.AD

Junior doctors can find the process of doing an audit helpful in gaining an understanding of the healthcare processheres how to do one
In the United Kingdom, doctors in the first two years after graduation are asked to perform an audit. Audit measures practice against standards. Unlike research (which asks the question, what is the right thing to do?), clinical audit asks, are we doing the right thing in the right way?1 Clinical audit forms part of clinical governance, which aims to ensure that patients receive the best quality of care. Clinical governance is often defined as how NHS organisations are accountable for continually improving the quality of their services and safeguarding high standards of care.2 Audit can include assessment of:  The structure of carefor example, resources such as the presence of a dedicated stroke unit  The process of carefor example, waiting times in clinics  The outcome of carefor example, blood pressure reduction in response to therapy. Audit should also be transparent. It should not be confrontational or judgmentalit is not an opportunity to name, shame, and blame.
Does audit work? There is conflicting evidence on whether audit works. Audit and feedback has not consistently been found to be effective.3 For every success story there is a project that has run into the ground without showing any substantial contribution to quality of services. The most frequently cited barrier to successful clinical audit is the failure of organisations to provide sufficient protected time for healthcare teams to participate.4 The most recent systematic review concluded that providing healthcare professionals with data about their performance in the form of audit and feedback may help improve their practice.5 However, in the trials included in the review, the effects varied widely, from an apparent negative to a very large positive effect. When effective, the effects were mainly small to moderate. The review concluded that the relative

effectiveness of audit and feedback was likely to be greater when baseline adherence to recommended practice was low and when feedback was delivered more intensively. However, the evidence presented in this review did not support mandatory use of audit and feedback as an intervention to change practice.5 It seems ironic that the tool used to measure whether we are doing the right thing in the right waywhere audit criteria should be derived from the best evidence is not itself supported by particularly strong evidence. So why do we persist in audit? As audit is part of clinical governance, to assess whether patients are receiving the best quality of care it is essential that we measure practice to know when we need to change it. Audit provides us with the best available tool to achieve this objective. An example of a success story is the national audit of stroke in the United Kingdom (excluding Scotland), which has improved quality of care for stroke patients across all three countries.6 Audit is included in the foundation programme to allow you, as junior doctors, to gain an understanding of how to obtain, maintain, and improve the services you deliver now and in the future. Performing an audit may also help you in your own learning and understanding of the healthcare process in a particular field. It may also allow you to contribute to constructing or refining a clinical protocol. Barriers to successful audit may have been a reason for the lack of effect of audit found in some of the trials included in the review mentioned above. So we all need

Stage 1: Preparing for audit

Stage 5: Sustaining improvements

Stage 2: Selecting criteria for audit review

This series aims to help junior doctors in their daily tasks and is based on selected topics from the UK core curriculum for foundation years 1 and 2, the first two years after graduation from medical school.

Stage 4: Making improvements

Stage 3: Measuring level of performance

The audit cycle

1241

BMJ | 31 MAY 2008 | VOLUME 336

PRACTICE

to try to overcome these barriers when we perform an audit.

How can you conduct an audit so that it is a success rather than a failure? If you are aware of the most frequent impediments to a successful audit, you can try to avoid them and make your audit a success rather than a failure. Good quality evidence on how to conduct an audit is lacking. However, the two elements that have been shown to be most effective in conducting an audit, based on evidence from a review7 and an opinion piece,8 are: an environment where audit is made a priority by the trust board, so that it is encouraged and supported; and the existence of a structured programme for audit, where a trust has a central clinical audit office that coordinates audit activity and brings together the results of audit for the trust as a whole. If these elements are not present, the audit is less likely to be a success. Most healthcare trusts should have a clinical audit lead. This person is responsible for creating and implementing a strategy for clinical audit, setting priorities for audit, and implementing an audit programme. Data for an audit are generally collected retrospectively. However, prospective data collection can give a team immediate feedback on its current performance and act as positive reinforcement to improve or maintain practice. Prospective audit usually requires good information technology resources.4 9 10 The audit cycle and spiral Clinical audit can be described as a cyclical or spiral systematic process (figure), with the ultimate aim of improving care. The spiral suggests that as the process continues, each cycle aspires to a higher level of quality. Stage 1: Preparing for the audit Identify problem and local resources for audit Selecting a topic for audit depends on the objectives of the audit and is likely to involve measuring adherence to healthcare processes that have been shown to produce best outcomes for patients. Consider also incorporating the views of all healthcare professionals involved in patient care, as in the national stroke audit.4 5 The clinical team has an important role in prioritising clinical topics. The following questions may help you select a topic:


Summary of elements of effective clinical audit*

    

Clinical audit should assess structure, process, or outcomes of care The audit should be part of a structured programme and should have a local lead Audit should ideally be multidisciplinary Patients should ideally be part of the audit Choose audit topics based on high risk, high volume, or high cost problems or on national clinical audits, national service frameworks, or NICE guidelines Derive standards from good quality guidelines Use action plans to overcome the local barriers to change, and identify those responsible for service improvement Repeat the audit to find out whether improvements in care have been implemented as a result of clinical audit Develop specific mechanisms and systems to monitor and sustain service improvements once the audit cycle has been completed

 

 

*Based on A Practical Handbook for Clinical Audit16

Have patients recommended topics? Patients priorities can differ markedly from those of clinicians. Practical approaches have been developed for involving patients in all stages of audit (including design), data collection, and implementing change.11 12 Is good evidence available to inform standards for example, systematic reviews or national clinical guidelines?

Is the topic a priority for the organisation? For example, have problems been encountered in any of the following areas? -High volumesuch as requests for chest x rays in the accident and emergency department -High risk to staffsuch as needle stick injuries in an HIV unit -High risk to patientssuch as certain postoperative complications -High costsuch as trastuzumab for breast cancer.

Locate relevant information  Where can you find clinical guidelines? From the National Institute for Health and Clinical Excellence (www.nice.org.uk); National Library for Health (www.library.nhs.uk); and Scottish Intercollegiate Guidelines Network (www.sign.ac.uk).  Where can you find criteria for clinical audit? From clinical guidelines or local hospital guidelines.  Where can you find information on service standards? -From national service frameworks, which are long term strategies determined by various stakeholders such as health professionals, service users, and managers to improve specific areas of care by setting measurable goals within set time frames (www.dh.gov.uk/en/Site map/DH_A-Z_AZSI). For example, the goal that no patient should wait longer than one month from an urgent referral by their GP with suspected cancer, to the start of treatment.13 -From the National Centre for Health Outcomes Development (http://nchod.uhce.ox.ac.uk) -From the Health Commission Wales (Specialist Services) (http://new.wales.gov.uk/topics/health/ hcw/?lang=en).  Which organisations have information about clinical audit? The royal colleges and other professional bodies. The Clinical Effectiveness and Evaluation Unit of the Royal College of Physicians (www.
BMJ | 31 MAY 2008 | VOLUME 336

1242

PRACTICE

Worked example: how to perform the perfect audit Stage 1: Preparing for audit Identify the problem and the local resources available for audit  Dr Black is a foundation year 2 doctor on a respiratory firm. One month into his attachment, he has noticed that patients with chronic obstructive airways disease who met the criteria for starting non-invasive ventilation had not been given this treatment. He discusses this with his consultant, who encourages him to do an audit.  Dr Black discovers that his hospital has a local audit lead and a clinical audit office. He contacts them and is told that they will be able to help him with the audit.  In agreement with his local audit lead, he chooses the British Thoracic Society guidelines* on non-invasive ventilation in acute respiratory failure and finds these on the societys website. Stage 2: Selecting criteria Determine what you are trying to measure and define gold standards Dr Black writes the following criterion statement, derived from the thoracic societys guidelines: To measure the percentage of patients with an acute exacerbation of chronic obstructive pulmonary disease with a respiratory acidosis (pH <7.35), despite maximum medical treatment on controlled oxygen therapy, who received treatment with non-invasive ventilation.  After discussion with the respiratory team, he chooses a standard of 90% as he thinks that in the given environment, as an optimum standard, this percentage is more realistic than an ideal standard of 100%.


Stage 3: Measuring level of performance Collect data He decides to collect data from the medical notes of the last 50 patients admitted with an acute exacerbation of chronic obstructive pulmonary disease. He asks the audit office to obtain the notes for him. He creates the following audit proforma (then it takes him two afternoons to collect the data): 1 Does the patient have a respiratory acidosis (pH <7.35) despite maximum medical treatment on controlled oxygen therapy? Yes/No 2 Were they treated with non-invasive ventilation? Yes/No 3 If they were not treated with non-invasive ventilation, was a reason given? Yes/No Compare performance with criteria  Dr Black analyses the data against the criterion statement. Forty of the 50 patients had a respiratory acidosis, despite maximum medical treatment on controlled oxygen therapy; 31 of these 40 patients (77.5%) were treated with non-invasive ventilation.  This percentage did not meet the standard of 90%, so he looked at the reasons why non-invasive ventilation had not been started in nine of these patients. In four patients the respiratory acidosis was noted but it was documented to continue medical treatment. In two patients the respiratory acidosis was thought to be a metabolic acidosis. In two patients no noninvasive ventilation machine was available, and both patients required intubation. In one patient, by the time the decision to start non-invasive ventilation had been made by the medical registrar, the patient had required intubation. Stage 4: Making improvements  Three months into his attachment, Dr Black presents the audit to the medical and emergency departments. The consensus is that the two main reasons that non-invasive ventilation was not used were lack of enough machines and lack of knowledge of medical staff on when to use it.  The clinical directors of the medical and emergency departments agree on the following action plan to improve the use of non-invasive ventilation in these patients: 1 Educate all medical staff on the interpretation of arterial blood gases and use of non-invasive ventilation. This education was to be delivered by tutorials on blood gas interpretation for all junior doctors twice yearly and training in non-invasive ventilation. The respiratory consultants and specialist registrars were to give these in the first week for all new starters. 2 Display posters on when to use non-invasive ventilation in patients with chronic obstructive pulmonary disease in the emergency departments and medical wards. The posters were to be designed by the clinical director of the emergency department and produced by the medical illustration department. The plan was to display them within the next three months. 3 Develop a business plan to increase the number of non-invasive ventilation machines in thehospital. This was to be drafted by the clinical directors of the medical and emergency departments and presented to the chief executive within the next three months. Stage 5: Sustaining improvements Repeat audit  After a year, although Dr Black is now in gastroenterology, he repeats the audit. The first two changes on the action plan have been implemented successfully, and the hospital now has three more non-invasive ventilation machines. He finds that 90% of patients are meeting the criteria.  Owing to the improvements, Dr Black and the respiratory consultant devise a monitoring tool to ensure that the improvements are sustained. The tool consists of a short checklist on an A4 sheet of paper. This checklist is attached to the notes of all patients with chronic obstructive airways disease. It provides a simple way to ensure that when such patients meet the criteria for treatment with non-invasive ventilation, this treatment is used appropriately. The tool can be used for retrospective or prospective data collection.  At the repeat audit, it may be appropriate to set the standard higher than 90%.
* The society plans to update this guideline, with work to update it starting in 2008

BMJ | 31 MAY 2008 | VOLUME 336

1243

PRACTICE

rcplondon.ac.uk/college/ceeu/index.htm) works to promote clinical standards through audits, guidelines, and related activities to improve health care.
Stage 2: Selecting audit review criteria You can use recommendations from clinical practice guidelines to develop criteria and standards. This could save you time and additional work.

Tips for busy clinicians



Before you start an audit, find out whether your hospital has a local audit lead. If it does, contact him or her for advice and information Choose a topic that reflects national clinical audits, national service frameworks, or NICE guidelines Use an audit proforma. This will make collecting the necessary data easier

 

Determine what you are trying to measure Audit criteria are explicit statements defining an outcome to be measured. They should relate to important aspects of care and be derived from the best available evidence. Having explicit selection criteria will ensure that the data you collect are precise and that you collect only essential information. Define ideal standards For the criterion to be useful, you need to define the standard (the level of care to be achieved for any particular criterion, which is usually expressed as a percentage).14 Ensure that the standard you choose is realistic for your given environment (see the box with a worked example). Anderson, in his ABC of Audit, writes: A minimum standard describes the lowest acceptable standard of performance. Minimum standards are often used to distinguish between acceptable and unacceptable practice. An ideal standard describes the care it should be possible to give under ideal conditions, with no constraints. Such a standard by definition cannot usually be attained. An optimum standard lies between the minimum and the ideal. Setting an optimum standard requires judgment, discussion and consensus with other members of the team. Optimum standards represent the standard of care most likely to be achieved under normal conditions of practice.15 Two examples of audit criteria (with a different standard defined for each criterion) are:  To measure what percentage of patients with septic shock were given anti-infective treatment. As septic shock is a condition with a high mortality rate, it would be appropriate to aim for a standard of 100%.  To measure what percentage of patients in the rheumatology outpatient clinic were seen within one hour. Here it may be acceptable for the standard for the first audit to be 80%.
Stage 3: Measuring levels of performance Collect data Some hospitals have audit teams that may help with data collection. An audit proforma is useful for collecting data and can be readily derived from established guidelines and protocols. You need to define the patients to be included and excluded in the audit, the audit review criteria, and the time period over which the criteria apply. The data may be available in a computerised information system, but it may also be appropriate to
1244

collect data manually depending on the outcome being measured. In either case, you will need to consider what data you need to collect, where you will find the data, and who will collect the data. Although clinical records are frequently used as the source of data, they are often incomplete. Collecting data from several sourcessuch as clinical records, blood results from patient administration systems, and imaging from picture archiving and communications systemscan help to overcome this problem. Electronic information systems are useful not only for collecting data but also for improving access to research evidence, prompting change through record templates, and introducing revised systems of care. Compare performance with criteria This is the analysis stage.


Compare the data collected with criteria and standards  Conclude how well the standards were met  If they were not met, identify reasons for this. In theory, if the standard was not met in 100% of the standard that was set, there is potential for improving care. Remember, the standard set may have been 90%. In practice, if the results are close to 100% of the standard, you may decide that any further improvement will be difficult to achieve and that other standards, with results further away from 100%, are the priority targets for action. However, this decision also depends on the topicin some life threatening situations, it will be important to achieve 100% of the standard.
Stage 4: Making improvements Implement change Data collection has no chance of making any impact unless you follow it up with the more difficult process of implementing changes. If you have completed stages 1 to 3 early in your four month attachment, you will be able to present your results within this time period. Once you have presented and discussed the audit results with the rest of the team and whoever else in the hospital the audit is relevant to, you must agree on recommendations for change. Use an action plan to record these recommendations, also indicating who has agreed to do what and by when. Disseminating educational materials, such as guidelines, has little effect unless accompanied by selected
BMJ | 31 MAY 2008 | VOLUME 336

PRACTICE

implementation methods, such as tutorials, reviews, or reminders.4

Stage 5: Sustaining improvements This stage is critical to the successful outcome of an audit: it verifies whether the changes implemented have had an effect and determines whether further improvements are needed to achieve the standards identified in stage 2.

KEY POINTS
Audit measures practice against performance The audit cycle involves five stages: preparing for audit; selecting criteria; measuring performance level; making improvements; sustaining improvements Choose audit topics based on high risk, high volume, or high cost problems, or on national clinical audits, national service frameworks, or guidelines from the National Institute for Health and Clinical Excellence Derive standards from good quality guidelines Use action plans to overcome the local barriers to change and identify those responsible for service improvement Repeat the audit to find out whether improvements in care have been implemented after the first audit

Repeat the audit Although as a foundation doctor you will not be able to complete the full audit cycle within your four month attachment, you will be able to collect and compare the data within this time period. However, to complete the cycle, after an agreed period, the audit needs to be repeated. If you wish to complete the audit cycle yourself, you could come back and complete the cycle. If not, you or your consultant must ensure that the cycle is completed by someone else otherwise your time spent on stages 1 to 3 of the audit cycle will have been wasted. You or whoever is going to complete the audit cycle (possibly the next doctor on the rotation) should use the same strategies for doing the audit to ensure the original audit is comparable. The repeat audit will hopefully show that changes have been implemented and improvements made. Develop tools to sustain improvements If these improvements are sustained, some form of monitoring should replace a full audit. The team should develop structures and systems that integrate, monitor, and sustain the improvements implemented as part of clinical audit. But if performance deteriorates, the full audit should be reactivated.
What are the challenges? The main challenge is how to make your audit a success. Firstly, you need support. Hopefully your trust will have a clinical audit lead; if not, ensure that your consultant is on your side and enthusiastic, otherwise achieving success may be difficult. Tell your consultant about a local audit presentation you attended where a repeat audit showed that patient care really had improved after the audit or cite an audit success story, such as the national stroke audit. Secondly, give yourself enough timeyou should probably start thinking about choosing an audit topic one month into your attachment. With access to a computer and case notes, in a retrospective audit you could collect the data within hours. Thirdly, be realistic when setting the standards for your audit. Dont choose a standard of 100% if you know that your trust cannot possibly meet this ideal standard. Aim for an optimum rather than an ideal standarddiscuss with the team what the optimum standard should be.
BMJ | 31 MAY 2008 | VOLUME 336

Finally, remember that audits are more likely to be effective when baseline adherence to recommended practice is low and feedback is delivered more intensively.5
This article is based on a previous BMJ Learning module by the same author (http://learning.bmj.com/learning/main.html). Contributors: AB is the sole contributor. Competing interests: None declared. Provenance and peer review: Commissioned; externally peer reviewed.

1 2 3

4 5

8 9

10

11

12 13

14 15

16

Smith R. Audit and research. BMJ 1992;305:905-6. Scally G, Donaldson LJ. Clinical governance and the drive for quality improvement in the new NHS in England. BMJ 1998;317:61-5. Grimshaw JM, Shirran L, Thomas R, Mowatt G, Fraser C, Bero L. Changing provider behavior: an overview of systematic reviews of interventions. Med Care 2001;39(suppl 2):II2-II45. National Institute for Health and Clinical Excellence. Principles of best practice in clinical audit. London: NICE, 2002. Jamtvedt G, Young JM, Kristoffersen DT, OBrien MA, Oxman AD. Audit and feedback: effects on professional practice and health care outcomes. Cochrane Database Syst Rev 2006;(2):CD000259. National sentinel stroke audit 2006. Prepared on behalf of the Intercollegiate Stroke Working Party. London: Royal College of Physicians, 2007. Morrell C, Harvey G, Kitson A. Practitioner based quality improvement: a review of the Royal College of Nursings dynamic standard setting system. Qual Health Care 1997;6:29-34. Fraser R, Baker R. The clinical audit programme in England: achievements and challenges. Audit Trends 1997;5:131-6. Dixon N. Good practice in clinical audit. A summary of selected literature to support criteria for clinical audit. London: National Centre for Clinical Audit, 1996. Kendall JM, McCabe SE. The use of audit to set up a thrombolysis programme in the accident and emergency department. Emerg Med J 1996;13:49-53. Kelson M. Promoting patient involvement in clinical audit. Practical guidance on achieving effective involvement. London: College of Health and the Clinical Outcomes Group, 1998. Balogh R, Simpson A, Bond S. Involving clients in clinical audits of mental health services. Int J Qual Health Care 1995;7:343-53. Department of Health. The NHS cancer plan: a plan for investment, a plan for reform. 2000. www.dh.gov.uk/en/Publicationsandstatistics/Publications/ PublicationsPolicyAndGuidance/DH_4009609. Irvine D, Irvine S. Making sense of audit. Oxford: Radcliffe Medical Press, 1991. Anderson DG. ABC of audit. Cleveland Vocational Training Scheme. www.pdptoolkit.co.uk/Files/Guide%20to%20the%20PDP/content/ audit.htm. NHS Clinical Governance Support Team. A practical handbook for clinical audit. 2005. www.cgsupport.nhs.uk/Resources/ Clinical_Audit/7@Appendix_1.asp

1245