Effective Auditing: How to Be a Good Auditor

Alan S. Bunker

Abstract The primary responsibility for ensuring that an audit is effective in serving its intended purpose rests with the auditor. This article describes how a good auditor plans and conducts an audit. The topics covered include the steps to take to prepare for an audit, establishing and maintaining a professional atmosphere, basic auditing techniques, the most likely areas for discovering problems, common auditing pitfalls, communicating results, and appropriate audit follow-up actions. Special sections are included on Auditor Roles, Audit Outlines, and Auditor Etiquette. Key Words: Management Auditing Methods Audit Reports

A JOB WORTH DOING RIGHT A dictionary definition of auditor might be one who examines with intent to verify. A more frequently heard refrain goes like this: those who can do, those who cant teach, and those who cant do or teach become auditors. Unfortunately, there is a good deal of truth to this refrain, and it stems from two important facts: (1) auditees often perceive the auditing process as unnecessary and counterproductive and (2) auditors often fail to perform their jobs in an effective and professional manner. In other words, auditing is a waste of time. Is this really true? Yesif the auditing process does not correct errors and stimulate improvements, then it is a waste of time. Yet, we know that every radiation protection program can benefit from periodic examination by objective outsiders, and every manager, supervisor, and technician stands to learn something of value when given the opportunity to share and compare knowledge, skills, and experiences with auditors. Audits help us measure our accomplishments without them, we dont really know where we stand. Auditing is necessary, and it is a job worth doing right! The primary responsibility for ensuring that an audit is effective in serving its intended purpose rests with the auditor. Management can establish the scope of an audit, and the auditee can cooperate in making his program and facility ready and available for inspection, but it is the auditor himself who will determine the success or failure of the audit. A good auditor will be able to identify both strengths and weaknesses in a positive manner that will be beneficial to the auditee and his management. A bad auditor will irritate the auditees personnel, depress

Editors Note: Although originally written for large facilities and nuclear power plant auditors, this article is also applicable to smaller operations. Auditing is a fine art, varies immensely among users, and almost always produces strain and anxiety within the auditee groupbut it is an essential task. Whether simply reviewing department procedures or inventorying materials, the concepts outlined here will prove valuable.

RSO Magazine Volume 9, No 3 2004

organization morale, and divert management attention to unproductive pursuits. It takes technical expertise, a professional attitude, and lots of hard work to be a good auditor. Anyone can be a bad auditor. Most of us are called upon to perform audittype functions at one time or another in our careers, either as official inspectors or evaluators, or as consultants or troubleshooters. To be effective in any of these roles, you must understand and apply the proper auditing techniques. If youre committed to being a good auditor, you can make the auditing process a valuable learning experience for both you and the auditee. Its a challenge: youll be pitting your experience and skills against the collective experience of a group of professionals and the intransigence of years of program operation. Your success will be determined by how well you prepare, how thoroughly you examine, and how well you communicate.

collect information for subsequent analysis by another individual or group, verify compliance with regulations or policies, verify adherence to procedures, identify problem areas (e.g., safety hazards, inefficiencies, recurring errors, etc.), identify excellent areas (e.g., outstanding individual or group performance, cost-saving measures, good facility design features, good operational practices, etc.), investigate an unusual occurrence or incident to determine facts, reconstruct events, and/or determine causes, or analyze a known or suspected problem area and make recommendations for improvements.

PREPARING FOR AN AUDIT The hallmark of a good auditor is preparation. When you arrive at the auditees site, you must know exactly what you are going to do and how you are going to do it. Your time in the field is too valuable to be wasted on unproductive preliminaries. Instead, you must plan every possible aspect of the audit in advance. Determine the Basis for the Audit Your approach will be influenced by the overall basis for the audit, that is, the why, the who, and the what of the audit. Before you decide how you are going to perform an audit, you should examine each of the following points: What Is the Purpose of the Audit? What you will examine and the role that you will play will be determined largely by why the audit is being performed. Sometimes it is very easy to lose sight of the original purpose of an audit once you have become involved in your examination. So you should begin by deciding which of the many possible purposes your audit will serve. For example, audits can be performed to:

What Is Your Role? Given the purpose or purposes of the audit, you must decide what type of perspective to adopt: data gatherer, judge, consultant, or salesman (see inset on Auditor Roles). Chances are that even in the simplest of audits, you will find it necessary to adopt one or more of these roles. You must, however, recognize when to make the transition from one role to another and when not to combine roles. For example, if your purpose is to investigate an incident, you will want to act strictly as a data gatherer and withhold your judgment of the situation and any comments you might have until you are sure that you have obtained all the relevant facts. Likewise, if your purpose is to verify compliance with regulations, you must be both a data gatherer and a judge, but you will want to avoid making suggestions (consulting) that might be interpreted by the auditee as indicating areas of non-compliance. Who/What Is the Authority? The authority for the audit determines what will be done with the results and who you, as the auditor, are responsible to. In theory, the purpose of any audit is to assist the auditee. But in reality, you must tailor the conduct of your audit and the final results that you produce to best serve the interests of the authority for the audit, be it a regulatory agency, an industry review group, an insurer, a Quality Assurance group, or the auditees

10

RSO Magazine Volume 9, No 3 2004

management. You must also keep in mind that what you should seek to audit and what you will be allowed to review will be determined by the nature of the authority. For example, if the authority is an insurer, you would be interested mainly in items that might represent undue risk of loss or liability, and you would expect to have access to all insured facilities and all relevant records. On the other hand, if the authority is a regulatory agency, you would examine only items within the agencys jurisdiction, and you would expect the auditee to restrict your access to nonlicensed areas and records. To summarize: dont forget who youre working for (youre not working in for the auditee). What Are The Standards? Determine what for you will use as a basis for comparison in your examination of the auditees program and facilities. If you will be using official, published criteria, be sure that you are thoroughly familiar

with the criteria and any and all policies regarding their interpretation and application. Be sure of your footingyou dont want to tell the auditee one thing and then have the authority cause you to have to change your story. Likewise, you dont want the auditee to find you lacking in your knowledge of the criteria (expect the auditee to know the criteria as well or better than you do). If you will be using unofficial standards, be sure that you know (and that you communicate to the auditee) the exact basis for your standards. Examples include your own experience and expertise, other programs and facilities similar to those of the auditee, and generally accepted industry practices. What Is the Scope? Decide in advance what items you must examine, what items you would like to examine, and what items you can forget. Be realistic about what you can accomplish in the time allowed. Restrict the scope to those areas that appear to offer the best prospects for accomplishing the purposes of the audit. If there are too many of these, give consideration to performing more than one audit. Once you are at the auditees site and have started your examination, you can expect to be able to further narrow and refine the scope as you eliminate those areas unworthy of in-depth review and discover new areas worth pursuing. Establish an Audit Schedule Contact the auditee and determine the best time or times to conduct the on-site portion of the audit. Be sure to take into consideration both what you need to see and the auditees operational restraints. If at all possible, let the auditee select the time(s) for the auditthat way his organization will be more committed to giving you the cooperation you need (invited guests get better treatment than party crashers). It is best to avoid situations where you may overtax the auditees organization (e.g., bringing a large group of people on site at the beginning of a busy period). Never schedule your audit to occur at the same time as that of another authority, even if the auditee requests it. Dont permit the auditee to rush you into visiting the site before you are ready.

AUDITOR ROLES A Data Gatherer reviews the available facts, figures, and statistics on a given subject and analyzes these to determine which are the most relevant to the purpose of the audit. He uses these to arrive at a factual description of the situation as it exists, not as it should or could be. He must be objective and perceptive. A Judge evaluates the relevant facts and compares them to some standard, either official (e.g., regulations or inspection criteria) or unofficial (his expectations). He then decides whether the item he is examining meets, exceeds, or falls short of the standard. He must be objective and fair. A Consultant analyzes a situation in light of existing constraints, management objectives, available technology, and industry experience. He then recommends a course of action that will achieve the desired result. He must be technically competent and creative. A Salesman persuades the auditee to accept the auditors interpretation of the facts. He convinces the auditee that a particular course of action will produce an outcome that will benefit the auditee and/or the industry. He must be knowledgeable and enthusiastic.

RSO Magazine Volume 9, No 3 2004

11

Decide with the auditee what hours you will be permitted access to the facility, and what hours key personnel will be available for interviews. Determine the auditees schedule for routine staff meetings and training activities that you may want to attend or that will impact on the availability of personnel. If appropriate to the scope of the audit, plan to cover at least one backshift so that you can see how things operate without the crunch of dayshift administrative functions. Impress upon the auditee that you are flexible and that you will change your schedule as necessary to obtain the most complete picture possible of his program. Have the auditee designate a single individual as your main point of contact, and agree upon a frequency for debriefings with this individual. The purposes of these debriefing sessions are to keep the auditee informed on your activities, to work out logistics problems, and to obtain any extra assistance needed. Dont agree to debrief more frequently than once a day or to debrief with separate individuals (i.e., dont make yourself responsible for communicating up the linelet the auditees personnel do that). Avoid debriefing with groups of personnelyou will be pulled in too many directions at once. Keep debriefings short and simple, and dont pester the point of contact with continuous updates on your progress. If you feel it will help, you can set up a schedule of what you plan to look at each day that you are at the auditees site. Be prepared to revise the schedule the first day, and every day thereafter. Dont give your schedule to the auditee. The preferred approach is to start with a list of the major areas that you want to examine along with the time you think it will take to cover each area. This will tell the auditee what to expect without giving him the impression that you want him to alter his operations to suit you. You and the auditee can then work from the list on a dayto-day basis as the audit proceeds.

What you learn will help you to refine the scope of your audit and give you clues on what to examine. It will also help you avoid wasting your time on subjects that have been adequately covered by previous audits. Your pre-audit review should include the following: Review Procedures. Get a copy of the auditees procedure index and use it to select samples of procedures that cover the planned scope of your audit. Obtain up-to-date information copies of these procedures and review them carefully to get the flavor of the auditees program. Remember, however, that you are auditing the program, not the procedures (unless verification of procedures is a specific purpose of the audit). Procedures are good road maps, but they dont take into account all the detours caused by road construction, traffic jams, and bad weather. When you visit the auditees site, you will no doubt find some differences between how things are done and how the procedures say they are done. This is not necessarily badit is simply a fact of life. Review Performance Data. Depending upon the scope of your audit, the auditee may be tracking and analyzing performance data that you can review to determine trends and to identify areas for further examination. Examples include person-rem expenditures, radiological incident/ occurrence reports, skin contaminations, dosimetry comparisons, radwaste volumes, qualification test scores, and personnel overtime. You should ask for these data whenever and wherever they are available, but you should not ask the auditee to prepare special reports just for your audit. This is a sore subject with some auditees because they feel that they will be giving ammunition to the auditor. Be sensitive to this point of view, but dont fail to ask for the data if you can put it to good use. Review Internal Reports. Ask about and obtain copies, if possible, of any relevant internal reports. Examples include budgets, project schedules, internal audit reports, ALARA reviews, system design reviews, special studies, and outage reports. Anything that the auditees management is using to monitor performance or evaluate planned improvements is fair game.

Perform a Pre-Audit Review With a little effort on your part, and early cooperation from the auditee, you can learn a tremendous amount about the auditees facility and program long before you set foot on site.

12

RSO Magazine Volume 9, No 3 2004

Review Previous Audits. Look at any available reports from previous audits (of the specific auditee). You will be most interested in those audits performed by or on behalf of your authority; however, other organizations viewpoints will be helpful. Keep in mind that the older the information, the less likely it is still valid. Also keep in mind the different political situations that may have influenced the outcomes of past audits. You should strive to be informed about these previous audits, but you should not let them influence your decisions when you conduct your own audit. Interview Other Auditors. Talk with those individuals in your organization that have examined the auditee before. Review their private notes, if available. This will help you to form a picture of the auditees past performance, but it obviously wont apply to the present situation. Treat all the evidence you collect in this manner as hearsay. Conduct Phone Interviews. Obtain a copy of the auditees organization chart and call the key individuals in the areas that you will be examining. Ask preliminary questions based on your procedure reviews and knowledge of past audits. Keep your interviews brief and recognize that individuals will be reluctant to discuss many subjects in detail over the phone. Conduct Surveys. In some situations, you may be able to gather relevant information and save on-site time by using preliminary questionnaires or exams. For example, if one of your purposes is to evaluate the effectiveness of the auditees training program, you might ask the auditee to administer an exam that you have prepared, a standard test, or one of the auditees own exams to a representative group of trainees or recent trainees. In another situation, if you are tasked with evaluating employee motivation, you would want to obtain information on employees perceptions of work conditions, supervisor relations, etc., with a questionnaire. Be absolutely certain to report the results of your analyses of such surveys/exams to the auditeeotherwise, you wont get his cooperation for similar surveys in the future. There is one source of information missing from this list that you might be tempted to use. If

you have friends that work for the auditees organization or that work or have worked for contractors of the auditee, dont ask them for their opinions or views. The information that you might obtain would be of little use, and you would undoubtedly be placing your friends in very awkward, if not unethical, situations.

Prepare an Audit Outline Draw up an outline of the areas that you plan to examine and list the elements that you feel are necessary to conduct a complete review in each area. The inset shows an example of a list that an auditor might use to review a radiation protection instrument program. Your organization may have prepared audit outlines and called them guidelines or checklists. You may also find that some published criteria are so specific that they can be used directly by you (or the auditee) as an outline for the audit. Whatever the case, be sure to custom tailor your outline for the specific audit at hand, as well as for your own style of auditing. An audit outline is for your use in planning your time in the field so that you cover everything in a logical manner and so that you dont forget to collect information that you might need later on to complete your audit report. Never show your outline to the auditeeif you do, it will automatically become the standard for the audit, replacing any official criteria that might apply. Dont use your outline as a checklist during an interview (review it before the interview), and dont read to the auditee from your outline. You can take notes directly on your outline if it helps you organize your thoughts, but keep in mind the fact that the outline will not dictate the relationships between the facts that you collect (in other words, dont base your conclusions on your outlinebase them on the facts). Keep revising your outline as the audit proceeds to make it fit the situation at hand.

Arrange Logistics Contact the auditee and make arrangements for any training, testing, and screening that will be necessary to gain entrance to the facility. Obtain

RSO Magazine Volume 9, No 3 2004

13

all necessary paperwork and check it to be certain that it is correct. You wont make much of an impression as a dosimetry system auditor if you show up on site without a Form 4 (exposure history) or with one that is filled out incorrectly. Ask the auditee to set aside a work area for your use during your visit, preferably with access to a phone and a complete set of the auditees procedures. EXAMPLE AUDIT OUTLINE: RADIATION PROTECTION INSTRUMENTS Interview instrument supervisor Obtain instrument inventory/status list for data file Inspect instrument storage areas (conditions, labels, OOS instruments, random checks for operability) Inspect instrument calibration facilityInspect instrument repair shop Inspect warehouse and local spare parts storage areas Observe an instrument calibration Observe a non-routine survey Review calibration scheduling system Review deficiency record system for trends Review sample model maintenance and calibration records for trends During facility tours, check in-use instruments (calibrated, right for job, operability) Interview technicians (good/bad instruments, most-used, unmet needs) Interview operators (HRA entry and rounds) Review instrument budget Track an instrument through a deficiency repair-calibration cycle (status I.D., communications, records, repair test before cal, scheduling) Exercise: pull an in-use instrument at or past mid-cycle for calibration and test in calibration facility

When you make arrangements for access to the auditees facility, you should give careful consideration to whether or not you need to obtain unescorted access. In some facilities, obtaining full clearance for unescorted access may require a day or more of training, testing, screening, filling in forms, and waiting. If your visit will be limited to only a few days, you will probably be better off going the escorted route, unless the scope of your audit prevents it (e.g., if you must enter areas of the facility, such as high radiation areas, where full security and radiation protection clearance is required). Talk this over with the auditee to decide exactly what level of access will be best for your audit. Ask the auditee to be prepared to supply you with knowledgeable escorts to accompany you on inspections and tours of the facility. Many auditors shun the idea of being escorted because they feel they wont have the complete freedom of access that they need to be effective. This belief stems from a basic distrust of the auditee and the expectation that something will be hidden from the auditor. There is no such thing as complete freedom of accessnot even the auditees own personnel are free to go wherever they want in their facility. If you think that you are going to be able to find some hidden catastrophe by wandering around the auditees facility unescorted, youre wrong. What you will find is that it is very easy to get lost and get into trouble when youre in an unfamiliar facility. You will see just as much, and you will understand more of what you see, if you let the auditee supply you with guides. As for hidden problems, if you pay attention and ask questions, you will be able to spot any areas that the auditee is deliberately trying to steer you away from.

Conduct a Self-Qualification Check Before you launch yourself full-speed into the field, take a few moments to be certain that you are qualified to perform the audit. Ask yourself the following questions: Do I have the technical background to be a competent evaluator in all areas of the audit? Do I have the experience necessary to deal with this particular auditee?

14

RSO Magazine Volume 9, No 3 2004

 Do I have sufficient backup (other auditors and staff personnel available to support me in areas where I could run into difficulty)? Can I be objective about this particular auditee? If you cant answer yes to all of these questions, youre not ready to perform the audit. Be particularly cautious about placing yourself in a situation where you will be at a severe disadvantage when dealing with the auditee, either by virtue of your technical knowledge or your experience. Also, be honest with yourself about your feelings towards the auditee, especially if you have worked for or audited the auditee in the past. SETTING THE TONE OF THE AUDIT You can establish the tone of an audit by projecting the correct appearance and attitude and by taking into account the auditees natural reactions to your presence. You should strive to create and maintain a professional atmosphere that will be conducive to the purposes of the audit.

impression that you do not have your full attention on the audit. Dont chew gum or tobacco either chewing interferes with your speech. Try very hard to arrive each day wide awake, alert, and looking refresheddont allow a hangover or lack of sleep to be readily apparent (change your schedule, if necessary). Dont use drugs to enhance your alertnessyoull only lower your efficiency, and your behavior may confuse the auditee.

Act the Part Conduct yourself in a business-like manner and maintain as low a profile as possible. Listen actively and avoid lecturing or speaking in a manner that might be construed as condescending. Keep your conversations to the point and avoid being excessively friendly or chummy. Dont joke about the audit or the auditee. Conduct your examinations discretely, and be polite and courteous to all of the auditees employees. Be certain to follow the basic rules of Auditor Etiquette shown in the inset.

Look the Part Wherever possible, conform to the auditees dress code. Dont show up for the first day of an audit at a power plant wearing a three-piece suit. Instead, wear comfortable work clothes that project a business-like image: plain denim jeans or work pants, a shirt with a collar (not a golf shirt), and work shoes (not tennis shoes or wing tips). Leave your fancy trimmings at homedont bring or use expensive pens, sunglasses, musical watches, imported briefcases, etc. In short, make your appearance fit the occasion. One other thingif youre auditing an operating nuclear station, dont wear permanent-press knit clothing (it has an affinity for gaseous radioactive isotopes) and bring an extra pair of shoes (just in case your first pair should become contaminated). Dont allow your personal habits to detract from your professional image. Dont smoke (especially a pipe) while interviewing, inspecting, or observingfiddling with smoking paraphernalia is distracting and gives the
AUDITOR ETIQUETTE DONT manipulate controls or operate equipment. DONT offer suggestions for action on a problem unless specifically requested to do so by the auditee. DO follow and adhere to all rules and procedures established by the auditee. DONT ask the auditee to deviate from or otherwise violate any procedure, rule, or regulation. DO leave the area or the facility when requested to do so by the auditee. DONT discuss the audit with personnel who are not involved in the audit.

RSO Magazine Volume 9, No 3 2004

15

Understand Your Effect As an auditor, you must understand and be prepared to deal with the effect that you will have upon the auditee. Despite all that you may do to put the auditee at ease, you must remember that you are still the enemy. You are a paid investigator who has come to spy on the auditee and make him look bad. The auditee will always be on the defensive, and although he may be cooperative, you should never expect him to fully believe or trust you. Your chief ally in dealing with the auditee is time. Dont expect immediate agreement with each and every conclusion that you draw concerning his program. Instead, give the auditee time to examine your conclusions relative to his own experience. As time passes, the auditees initial reflex reaction to your presence will subside, and he will be more receptive to what you have to say. As you inspect facilities and observe activities, keep in mind the fact that your presence will influence the way that the auditees personnel perform their jobs. Some will be nervous and will make mistakes that they would not normally make. Others will be overly cautious and will take much longer to do the routine aspects of their jobs than normal. Many individuals will lose some of their normal self-confidence in the presence of an auditor, giving the appearance that they are not competent in their jobs. Some will do just the oppositethey will show off for an auditor and exceed the limits of their capabilities or authority. Your very presence upsets the normal balance of things, so take this into account when evaluating what you see and hear. Start Off On the Right Foot Regardless of how well you may know the auditee and his facility, always begin the onsite portion of your audit with an entrance meeting. Let the auditee determine who should attend the meeting, but be sure to include your main point of contact for the audit. Discuss your plans for the audit, especially any changes or new items that may have come up since you arranged for your visit. If the auditee has made any concessions or special arrangements for your visit, be sure to thank him for his cooperation. If the auditee wants

to give you a guided introductory tour of his facility, dont turn him down (especially if upper management will be conducting the tour). Dont be in too much of a hurry to start what you consider to be productive workgive the auditee time to adjust to your presence, and give the auditees upper management a chance to demonstrate their support for the audit. CONDUCTING THE AUDIT The secret to conducting an effective audit is knowing how to look, where to look, and what to look out for. Basic Auditing Methods There are six basic methods of gathering information in an audit. You should use each as it best suits the type of information that you want to obtain, but you should never rely on just a single method. Interviews. Always be certain you are talking with the right person for the information desired. Avoid group interviews (either more than one auditee or auditor)a one-on-one discussion is easier to focus and control. Keep your questions open-ended and guide, but do not dominate, the discussion. Interview whenever and wherever you get the chanceas you inspect and observe, as you review records, as you conduct exercises, and even as you debrief the auditee. Inspections. As you inspect, proceed from the general to the specific. First get a good overall view of the facility, room, piece of equipment, etc., then examine prominent features, and finally, zero in on specific items that strike a discordant note. Ask questions as you inspect, beginning with very general inquiries on purpose and function, and then more specific ones on how the item is used. Keep your eyes and ears open, and avoid focusing too tightly on any single attribute, such as cleanliness. Records Reviews. A random sampling of the records for a given area (e.g., personnel exposure files) will give you an overall picture of the quality of the clerical aspects of record keeping, and sometimes it can yield clues on problem areas

16

RSO Magazine Volume 9, No 3 2004

(e.g., a surprisingly small number of whole body count reports might lead you to investigate certain aspects of the internal dosimetry program). Usually, however, any sample that you can take in a reasonable period of time will be much too small for firm conclusions. A more productive approach is to select records based upon a common characteristic that you suspect might be a problem area. For example, if you noted several out-of service instruments of one model type in your facility tours, you could review the calibration schedule and some of the instrument history files for that model to learn more about a potential recurring problem with maintenance or calibration. Observations. Often the easiest way to learn how a program actually works is simply to watch it work. Many routine activities and special operations will provide opportunities for you to see personnel working together in as close to normal circumstances as you are likely to find. For example, if you wanted to examine contamination control practices, you could stand in an out-of-the-way spot by an active contamination control point, where you might be able to see ten to twenty people process in and out, trash and tools surveyed, packaged, and removed, a health physics survey of the area, and a work crew briefing, all in only one or two hours time. You can ask questions about what you see, but be careful not to interfere and cause personnel to do something they would not normally do. Vertical Tracking. Sometimes referred to as vertical auditing, this consists of following a specific evolution from beginning to end, normally by tracking through the records that are created in the process. For example, in an audit of an exposure control program, you might pick a specific Radiation Work Permit and trace it from the time it was first requested by a scheduler or maintenance supervisor until a final post-job review was completed, examining each supporting record along the way: survey forms, instrument calibration sheets, dosimetry inputs, work/ access logs, ALARA reviews, training records, etc. It is easy to see how such tracking can lead you into areas outside your original scope; however, vertical tracking does let you see how all the different parts of a given program work together.

Exercises. Normally, the term exercise conjures up thoughts of large-scale drills that are run to test the function of emergency plans and to practice responding to plant casualties. Individual auditors can also run exercises on a much smaller scale to test an auditees programs, personnel, and equipment. For example, a dosimetry exercise might consist of spiking some badges in the onsite calibration facility and then submitting them for normal processing by facility or vendor personnel. Oftentimes, such exercises are tests that the auditee normally performs on a routine basis; however, as the auditor, you pick the times and circumstances for the tests. Dont run an exercise without the knowledge and cooperation of the auditeeit will undoubtedly backfire (e.g., dont walk on site with radioactive sources in your pockets to test the portal monitorsyou may violate conditions of the auditees material license and you may create a security infraction). Always attempt to support your conclusions with information gathered by as many different methods as possible. In other words, verify the information that you obtain by examining enough sources. Dont knowingly restrict yourself to only one viewpointif you do, your information will be distorted and the conclusions you reach will probably be wrong. For example, if a supervisor tells you in an interview that he has had problems with the dosimetry vendor, dont stop there. Follow up by reviewing dosimetry processing, quality control, and TLD/DRD comparison records, by observing a dosimetry changeout, and by running an exercise with some spiked TLDs. You may find that the problem is not the vendor at all, but an error in the quality control testing procedure.

Taking and Reviewing Notes A good auditor must have a well-organized approach to taking, reviewing, and refining his notes. During interviews, you should take written notes as unobtrusively as possiblerecord key points, but dont be elaborate. Your audit outline can be very useful for note taking, especially if you have taken the time to jot down some of the questions that you plan to ask. At the earliest opportunity after an interview, you should go over your notes and fill in any details that you did not

RSO Magazine Volume 9, No 3 2004

17

have time to record. Dont tape record interviewsyoull alienate the auditee and youll find that you yourself wont do as good a job at interviewing when you arent forced to take notes. During observations and exercises, try to avoid the appearance of being overly concerned with your note taking. Each time that you write something down, the auditee will think he has done something wrong. If you are writing continuously, the auditee will become nervous and frustrated by your silent judgment of his actions. Limit your note taking to key items only, then fill in the details from memory at a convenient time afterwards. Another occasion for a good memory arises whenever you tour or inspect contaminated areas. You cant take notes while you are dressed in protective clothing, so youll simply have to develop a mental note taking technique. One trick that works well is to briefly review what you saw with your guide after you are back in street clotheswith his help, you should be able to reconstruct the most important points. The notes that you do take are worthless unless you use them to organize your thoughts and to help you reach conclusions about the program you are auditing. You should review your notes at the end of each day, compare them to your audit outline, and rewrite or expand upon the notes as necessary to get the facts to fall into a logical pattern. Each time that you do this you are, in effect, replaying the audit in your mind -- you will discover new relationships between the facts and you will find out where you need to obtain more information to complete your analysis of the facts. Auditor Assistance If you are fortunate enough to have other auditors with you during your visit to the auditees site, be sure to take advantage of their experience and their different viewpoints. Discuss your progress with them and get their ideas on the significance of what you have found and the approaches that you can take to further evaluate areas and reach conclusions. Alert them to information that you need that they may come across in their own investigations. If you report back to a main office during a break in the audit,

you can run your preliminary conclusions by some auditors then and get additional input. But remember, while other auditors opinions are useful, you are still the one who has to do the work, and you are the only one who is responsible for your conclusions.

Easy Targets There are certain aspects of any program or facility that are more subject to developing problems than others. When your audits purpose is to identify problem areas, you can aim at the following targets and be fairly certain of finding something worth pursuing. Complex Programs. Any program, procedure, or system of getting things accomplished that appears to be quite complex normally has some unofficial shortcuts associated with it (counter to what management may believe). In the worst cases, these shortcuts will involve serious departures from approved procedures and possible unsafe practices. In the best cases, the shortcuts may be beneficial improvements that should be implemented officially. Old Procedures. Check the approval and revision dates on important written procedures. If a procedure has not been revised within the last two years, or if it was written and approved by individuals who are no longer on the auditees staff, the chances are good that the procedure is out of sync with the program. The significance of this problem will depend on how much reliance is placed upon written procedures in the auditees programs. Department Interfaces. There are always rivalries and communications problems between departments on the same site and between corporate and plant groups. These often lead to inefficiencies in any work that requires the cooperation of the different groups. Unrealistic Requirements. A policy, procedure, or method that appears to be too strict or lax in relation to its objectives usually warrants further review to establish its merits. For example, if your pre-audit procedure review revealed that radiation surveys were required every four hours in an access control area, your first impression

18

RSO Magazine Volume 9, No 3 2004

would be that this was excessive. Your audit might reveal that the requirement was valid at one time, but that it had not been changed to reflect current conditions, or that the requirement was an over zealous response to a previous incident. Old Facilities and Equipment. Radiation protection programs change over time, and normally facilities and equipment are upgraded to reflect these changes. Any facility, instrument, or piece of equipment that appears old in relation to current technology may indicate a problem area. For example, if, during in-processing, you noticed that the whole body counting system was based on an old-style, hard-wired MCA that you didnt recognize as having seen anywhere else, you would suspect it as being a source of problems. Your audit might show that the system was not capable of performing the types of analyses the radiation protection group should have been doing and that it was a bottleneck in the internal dosimetry program due to low throughput capacity. Temporary Fixes. Stopgap measures and jury-rigged equipment tend to become permanent fixtures of a program and can exacerbate the original problems that they were designed to fix. When you come across a situation that the auditee claims is temporary, find out just how temporary it really is and whether or not the original problem has been solved or merely hidden. For example, if you discovered that a temporary counting facility had been set up outside the normal controlled areas of the plant, a further investigation might show that the reason for this was an unexplained increase in background radiation levels in the main counting room or frequent gaseous activity releases within the building.

person and collect information by more than one method. Sometimes this can be frustrating, and you might feel like you are going in circles without getting any answers. Getting Only One Side of the Story. Dont restrict yourself to examining only the radiation protection groups version of the story. Check their suppliers and customers also. For example, if youre reviewing a radiation protection instrument program, youll need to check the instrument and control group (interface for repair work), the warehouse (procurement and spare parts), and instrument users other than radiation protection personnel, such as operators (end user problems). Running Down Dead-End Streets. Dont waste your time examining areas that are clearly outside the scope of your audit or that of your authority. Likewise, dont bother with those areas that are obviously not going to yield any significant results. Once you feel that you have collected sufficient evidence on a subject to support your conclusions, stop and move on to the next item. Dont let the auditee lead you into an unproductive quest for data that you dont need and wont use. Believing a Chronic Complainer. Eventually you will come across an individual who has been eagerly awaiting a chance to complain about the ills of his organization. These axe-grinders are easy to spotthey tend to have inflated opinions of themselves and they are almost always misunderstood by their supervisors and co-workers. Be suspicious of anyone who is overly anxious to volunteer damaging information. You should, however, search for the grain of truth in what he has to say by asking for specific examples and by verifying these with information that you obtain from other sources. Working a Pet Peeve. It is all too easy to examine only those areas that you particularly like to review or for which you are uniquely qualified. Likewise, it is easier to repeat the same routine from audit to audit than it is to create a special routine for each audit. Dont let your predisposition or past auditing experience dictate your approachtreat each audit as a separate and unique case.

Some Auditing Pitfalls There are many traps that await the unwary auditor in the rush to complete the full scope of an audit. Be sure to guard against the following pitfalls: Expecting Complete Knowledge. Not every individual on the auditees staff is or should be knowledgeable in all aspects of a program. If you expect to obtain a complete understanding of any program, you will have to talk to more than one

RSO Magazine Volume 9, No 3 2004

19

Jumping to Conclusions. Dont pass judgment until you are reasonably sure of your facts. When you do reach a conclusion, fly it by the auditee in a debriefing session to get his reaction. You may be surprised to find that you have misinterpreted what you saw or that you have forgotten to examine a crucial piece of evidence. Getting Too Involved. Dont allow yourself to become a part of either the problem or the solution. Once you stop auditing and start doing, you lose your objectivity. There are limits to the amount of consulting that you should or can do as an auditor. If you make suggestions, be sure that the auditee understands that they are just that suggestions only. The auditee is responsible for his program, not you. Failure to Provide Feedback. The auditing process is a two-way street: the auditee supplies you with information, and you are supposed to give him the benefit of your judgment in return. Keep the auditee informed of your basic impressionstell him what you think is good, what you think is bad, and why. Ridiculing the Auditee. No matter how stupid the error, never openly ridicule the auditees program or any of the auditees personnel. Keep emotion out of your criticism make it constructive, not destructive. Lobbying for the Auditee. Some auditees will try to influence your conclusions to support points that they may be trying to win with their management or with another auditing organization. They may even go so far as to supply you with comprehensive facts and figures to support a specific conclusion. Dont accept the auditees interpretation of the factsdraw your own conclusions. If you agree with the auditee, then fine, but dont go out of your way to represent the auditee. Accepting the Auditees Word. Never accept any type of verbal assurance or gentlemens agreement from an auditee over a problem area. The auditee may have good intentions at the time, but he also may not be able to obtain the management support necessary to fix the problem if you omit it from your audit report.

Finishing Without a Fight. Be very cautious of an overly acquiescent auditee. If the auditee does not raise any serious objections to your conclusions during the audit, he may be waiting until the final exit or report presentation to discredit you. Dont fail to get his honest reactions to your conclusions before you finish the audit.

Dealing with Problem Auditees Occasionally you will come across an individual who is extremely defensive and uncooperative. There are three possible approaches to dealing with such an individual: empathize with him and attempt to gain his trust, go around him by dealing with his coworkers or subordinates, or pull rank on him by pointing out his uncooperative attitude to his superiors. As long as such individuals represent isolated cases, they should not have any appreciable effect on your ability to conduct an audit. If, however, an uncooperative attitude is being fostered by top management, you may find it necessary to obtain assistance from the authority for the audit. Dont attempt to conduct an audit in an overly hostile environment. Instead, let the authority negotiate for a more reasonable approach on the part of the auditee.

Learning from the Auditee Every audit presents a golden opportunity for learning. Each program that you review and each facility that you visit allows you to expand your knowledge and to experience different ways of doing things and different points of view. Dont waste these opportunitiesbe alert for every new (to you) piece of equipment, procedure, technique, etc., and collect as much information as you can for your use in preparing for future audits (or for the time when you go back to doing or teaching). Most auditees will be very willing to supply you with details on what they consider to be the best aspects of their program and facility. However, you should respect an auditees refusal

20

RSO Magazine Volume 9, No 3 2004

to provide information for use outside the stated purposes of the audit, especially if any of the following situations apply: the program or facility is new or experimental, the information is proprietary, the work was performed by a contractor, or management policy precludes release of the information for use by outside parties.

Exit Meetings Whenever possible, present your preliminary findings to the auditee and his management in an exit meeting, either at the end of your visit to his site, or shortly thereafter. Encourage the auditee to discuss the findings, to ask any questions and raise any objections he has, and to give you his interpretation of the significance of each. Gauge the auditees reactions and those of his management carefullythey will give you the clues that you need to prepare your final report. You dont necessarily need whole-hearted acceptance of your findings by the auditee. Instead, you need to know which findings are the sensitive ones, which findings will require special documentation in the final report, and which findings you may need to re-evaluate to be sure that you have drawn the correct conclusions. An exit meeting is extremely important to the success of any audit because it represents perhaps the last opportunity that you will have to directly communicate your conclusions to the auditee. Your final, written report will be a distillation of what you found, and after all of the necessary word engineering by your organization, it may not convey the true essence of the audit. The exit meeting gives you a chance to deliver an unfiltered message to the auditeea message he can understand. The auditee needs this unfiltered version of your conclusions so that he can interpret the final report correctly and so that he can protect himself against inappropriate actions by others who might misinterpret the report.

COMMUNICATING THE RESULTS OF THE AUDIT Now comes the hard parttelling the auditee, the auditees management, your organization, the authority for the audit, and any other interested parties what you actually found. You must be able to clearly communicate your conclusions to sometimes diverse groups, both orally and in writing. You must also be able to sell your conclusions to the auditee and your own auditing organization. Your goal should be to provide sufficient information and to interpret the information as necessary so that every party to the audit will be able to take proper actions, whatever they might be (e.g., rewards, punitive measures, corrective actions, criteria changes, etc.). The term finding, as used in this discussion, refers to a formal conclusion of an audit. It could be a commendation for something that was considered to be excellent, a description of a problem in need of correction, a recommendation for improvement, or merely a statement of the facts as they were found. Some findings require a response by the auditee, either to specify and commit to a course of action, or to comment on a recommendation made by the auditor. Other findings may be presented for information only and may not require a response or specific action on the auditees part. Regardless of the response required, each finding should be firmly supported by the evidence collected in the auditit takes only one invalid finding to place all of the results of an audit in question.

Bad Findings There are some types of findings that simply should not be made, either because they will run counter to the purposes of the audit or because they could be misinterpreted by the auditee or his management. These bad findings are: Oral Findings. Never agree to limit your report on an audit to an oral presentation of the findings. Back your statements up in writing or have a transcript of your presentation made. Findings Without Standards. A finding that is not based upon the official or unofficial standards of the audit in some manner is a finding

RSO Magazine Volume 9, No 3 2004

21

that falls outside the scope of the audit and that of the authority. It is dangerous to comment on subjects that you are not qualified or authorized to review. Counteractive Findings. It is possible to compliment an auditee and cause harm to his program at the same time. For example, commending the auditee for his progress in cleaning up contaminated areas at a time when he is negotiating for additional decontamination funds with his upper management could cause his management to lower the priority assigned to decontamination in future budgets. Non-finding Findings. Never produce separate unofficial findings for delivery to the auditees management or any other party. These under the table comments can come back to haunt you and the auditing authority, especially if they cover an area in which the auditee later develops a serious problem. Findings Against Personnel. Unless the purpose of the audit is to judge the competency, promotability, etc. of specific individuals, you should not use the names of the auditees personnel in your findings. Referring to specific individuals by name may divert managements attention from the main point of the finding. Concentrate on the job position, not the individual who fills the position. Winning Auditee Acceptance Your daily debriefs during the audit should have prepared the auditee for the majority of your findings, and hopefully they will also have persuaded him to accept them as being valid. There will, however, still be disagreements, either over specific findings or the standards that they are based upon. Here are three techniques for selling the auditee on the validity of a contested finding. Review the Facts. Rerun the audit process that caused you to arrive at the finding. Show the auditee the specific examples in his facility and program that support the finding. This can be both enlightening and embarrassing for the auditee. Compare to Others. Show how the finding is supported by successful aspects of programs at

other facilities that are similar to the auditees. If possible, have individuals from these facilities contact the auditee and discuss their programs with him. Involve the Auditee. Get the auditee involved in an industry initiative related to the finding (e.g., working group, committee, peer review group, etc.) or in the auditing process itself as an observer or participant in a future audit. Rating Auditees The most frequently asked question in an exit meeting is: How does my program rate compared to others that you have seen? The best response to this question is to summarize the strong and weak points of the auditees program without giving an overall rating of the program. If pressed for a rating, you can state your personal impression in terms of good, average, or poor, but you should avoid direct comparisons to other auditees. There are simply too many variables involved to make legitimate comparisons between radiation protection programs using subjective ratings. The only legitimate comparisons are those based on measurable characteristics (e.g., number of personnel served, person-rem, radwaste volumes, etc.). There is one type of subjective rating that you should perform as a self-check on the completeness of your audit and the validity of your conclusions. When you have arrived at your preliminary findings, ask yourself what you feel the overall level of performance is for each major area that you examined. If your personal ratings dont match up well with the number, type, and significance of your findings, its time to look back and ask yourself why. You should feel comfortable with the findings that you are going to makeif you dont believe in them, you arent going to be able to convince anyone else of their validity. Report Writing While not as important to the auditee as the exit meeting, the final written report of the audit does serve as the official record of the audit, and therefore, you should ensure that it is as accurate

22

RSO Magazine Volume 9, No 3 2004

and complete as possible. Your auditing organization will most likely specify the exact contents and format for the reportyour input will probably be limited to providing a first draft along with supporting notes that may or may not be included in the final report. Here are some points to keep in mind as you struggle through the writing, review, and rewriting process: Write for your audience, not for yourself. Avoid being too technical (managers must be able to understand the report). Place every finding and statement of fact in the proper context to avoid misinterpretations (especially by uninvolved or uninformed parties). Avoid superlatives. Use adverbs and adjectives sparingly. Keep emotional statements out of the report. Support every finding with at least one example drawn from the facts determined in the audit. Dont stray too far from the original findings presented in the exit meeting. Be certain that your private notes support the written report. If they dont, make notations (in your notes) explaining why. Never prepare more than one version of the report (i.e., dont write one version for internal use and one for external consumption).

and your organization, and therefore, you may need your personal notes to back up your statements. Review the problems you may have encountered during the audit and decide what remedial action should be taken. For example, if you had difficulty convincing an auditee of the validity of some of your criteria, go over the bases for the criteria with others in your auditing organization and the authority for the auditsome revisions may be called for. If you felt unprepared to review any areas of the auditees program, go back over these and decide what action is necessary to be fully prepared in the future (i.e., training, a better pre-audit review, more industry data for comparison, etc.). In almost any audit, you will receive requests from the auditee for additional information or help. You should not commit to providing any specific assistance during the audit because you cannot be certain if you will be able to deliver on your promises. However, once the audit is complete, you should prepare an action list of items that the auditee requested and go over the list with your management to see which items you or someone else in your organization can fulfill. Get your managements commitment for the important items on the action list, and deliver the information or assistance as soon as possible. A few months after the completion of the audit (and after the auditee has received the final written report), call your main point of contact in the auditees organization and check on the progress that has been made in adjusting to the audit findings. Ask what parts of the audit were the most useful to the auditee and what parts caused the most trouble. Listen carefullysince the initial reaction to the audit should have subsided, you should take to heart what the auditee has to say at this point. Smooth any ruffled feathers that you canyou or one of your fellow auditors will probably visit the same auditee again in the future.

AUDIT FOLLOW-UP When youve completed your input to the report-writing process, collect all of your notes together and make a complete file package. Read over your notes one last time to be sure that they are complete (ask yourself if youll be able to understand them a year or two from now). Put your personal audit file in a safe place where you will have access to it in the future. If you submit the package to your organizations records management system, be sure to keep a copy for yourself. This package is your record of the audit, not your organizations. In the future, you may be called on to reconstruct your actions during the audit. Your recollection of the facts may differ from the official positions of the audit authority

CONCLUSION Audits serve useful purposes when they are performed correctly. The success or failure of any audit depends upon the qualifications of the

RSO Magazine Volume 9, No 3 2004

23

auditor and the techniques that he uses. Being technically competent is only a beginningto be a good auditor, you must also develop skills as a data gatherer, a judge, a consultant, and a salesman. You must prepare thoroughly for each audit, project a professional image, apply effective auditing techniques, avoid potential pitfalls, and communicate your conclusions clearly. Effective auditing is an acquired skillone that requires lots of hard work.

RSO Magazines Sponsors

All material published in these pages is copyrighted by RSA Publications, a division of Radiation Safety Associates, Inc., 2004. All rights reserved. Sponsors and advertisers make these pages possible; please support them (click below to visit their websites). Bicron Direct (www.bicron.com) Canberra (www.canberra.com) Perkin-Elmer (www.perkinelmer.com) RSA Laboratories (www.radpro.com/analysis) RSA, Inc. (www.radpro.com)

The Author Alan S. Bunker is the President of The Techrite Company and was the Founding Editor of Radiation Protection Management. His experience in applied radiation protection programs includes six years with the U.S. Navy as an Engineering Laboratory Technician and Radiological Controls Shift Supervisor, three years as the Radiation Protection Supervisor at Consumers Power Companys Midland Nuclear Station, and three years as a Project Manager with the Institute of Nuclear Power Operations. While at INPO, he evaluated health physics and chemistry programs at 17 power reactor sites in the United States, and it is this experience as an auditor that forms the basis for this article. Mr. Bunker holds a Bachelor of Business Administration Degree from Kennesaw College and has written extensively on applied health physics and management topics.

24

RSO Magazine Volume 9, No 3 2004