Beruflich Dokumente
Kultur Dokumente
Quality system
auditors
attitudes
39
Received May 1995
Revised September 1995
A.D. Vella
University of Luton, Luton, UK
Introduction
Quality assurance and quality management have become fashionable items for
the business and popular press[1-3]. One of the main topics of this interest,
especially in the UK, has been the ISO 9000 series standards for quality
assurance (the standards used for auditing are [4-6]), and the value and
significance of registration by third parties to these standards. This registration
is managed by a process of quality system auditing.
A quality system audit can be defined as:
A systematic and independent examination to determine whether quality activities and
related results comply with planned arrangements and whether these arrangements are
implemented effectively and are suitable to achieve objectives[7].
IJQRM
13,8
A set of principles and methods organised as a comprehensive strategy with the goal of
mobilising the entire company in order to achieve the greatest client satisfaction at the lowest
cost[11].
40
The willingness and ability that emerges when a company instils a sense of responsibility in
all the actors and secures their consent, especially at the highest level[11].
Not all these attributes of quality are easily determined from objective evidence,
and as quality assurance standards have evolved to take account of some of
these concerns (as a perusal of earlier quality assurance standards and early
versions of BS 5750 demonstrate), auditors are having to examine a wider range
of evidence in order to decide compliance.
Another issue is that the role of quality system auditing is changing.
Traditionally, audits were demanded by customers of their suppliers and
carried out by the customer or by an independent representative acting on their
behalf. However, as certification has become more popular, the benefits of
operating a compliant quality system have increasingly been sold to the
suppliers themselves. Their objectives for the quality system will be somewhat
different from those of their customers (although, of course, there will be
considerable overlap in aims). As a consequence, more stakeholders are
concerned with the results of an audit and the audit process must accommodate
the goals of these stakeholders. Because of all these questions, it was decided to
survey auditor attitudes and methods, to see how auditors currently address
these issues.
Design and execution of survey
Surveys are based generally on either interviews or questionnaires[12,13]. It
was decided to use questionnaires because there are over a thousand auditors
working for certification bodies and it would be time-consuming and expensive
to interview a significant fraction of them. The main difficulties with this
approach are that the range of questions that can be asked is limited and
clarification or follow-up of answers is extremely difficult (especially as the
questionnaire was anonymous). In the questionnaire, most of the questions were
closed (a closed question is where the auditor answers from a list of options,
rather than prepare an answer themselves), as this quickens the analysis and
presentation of the results.
In 1993 there were over 30 accredited certification bodies[14], but most of
these register companies in specific industries (such as the reinforcing steel and
ceramic industries). It was decided that, because of their size and specialization,
it would not prove useful to sample these companies. The certification field has
several large companies which offer a wide range of expertise: BSI QA, Lloyds
Register QA, SGS Yarsley, Bureau Veritas QI and NQA. Many of the auditors
working for certification bodies are employed by these companies so it was
decided to limit the survey to these companies. All these certification bodies
were contacted and agreed to participate in the survey.
Quality system
auditors
attitudes
41
IJQRM
13,8
Other
Produce quality products
Fulfil customer's needs
42
Figure 1.
Question 1, What do
you believe is the main
function of a quality
management system?
10
20
30
40
50
60
70
80
90 100
Number of replies
guarantee quality products represents few auditors opinions, only 3 per cent of
replies agreeing with this viewpoint (although it may be argued that product
conformance is the result of a controlled or improving quality system). These
answers reflect how auditors perceive that management (in particular) can
improve the operation of the company, and take a view of quality in keeping
with the interpretation of a quality system as a management tool.
Question 2a asked What do you personally consider are the most important
three characteristics that must be present in order that a company may have an
effective quality management system? A list of answers was presented (see
Figure 2) and the auditors asked to choose three.
Auditors believed that senior management commitment was the most
important attribute of a quality system, with most of the other replies receiving
between 25 per cent to 40 per cent support. Two attributes which few auditors
considered important were low levels of product non-conformance and status of
quality personnel. In view of the replies to question one, it is not surprising that
Figure 2.
Question 2a, What do
you consider are the
three most important
characteristics that
must be present in an
effective quality
management system?
Other
Status of quality staff
High product quality
Communication
Company environment
Management commitment
Audits/corrective actions
Understand customers
Committed to improvement
Trained staff
Documented procedures
0
50
Number of replies
100
150
200
250
0
50
Number of replies
100
150
200
250
Quality system
auditors
attitudes
43
Figure 3.
Question 2b, Which
clauses of ISO 9001
would encompass these
characteristics
mentioned in question
2a?
IJQRM
13,8
44
Figure 4.
Question 3, Compliance
to which three clauses
of ISO 9001 are the most
important in proving
the company operates
an effective quality
management system?
0
20
40
60
Number of replies
80
100
120
140
160
180
200
The answers to this question showed the importance auditors place in the
quality loop, with 71 per cent of the auditors identifying internal audits, 67 per
cent mentioning corrective action and 53 per cent of the auditors identifying
both activities. Twenty-nine per cent of the respondents answered with all three
clauses of the quality loop. However, management commitment was a
significantly less popular answer (from question 2), changing from 243 to 125.
Other characteristics highlighted in question 2b were also less popular; indeed
the number of replies for training dropped from 133 to 10. The popularity of the
quality loop activities for providing proof of compliance suggests that auditors
consider it to be a fundamental activity of the quality system and that it more
readily furnishes evidence of compliance than many other important activities
(although the auditor requires evidence of compliance of all activities). These
points are addressed in later questions.
Question 4a asked What do you believe are the three most important types
of evidence you can use in establishing compliance to ISO 9001? A list of 13
options was presented (see Figure 5). The purpose of this question was to
establish what evidence auditors look for while auditing, in particular whether
they rely solely on documented evidence or used more subjective evidence, such
as their perceptions of attitudes and opinions.
A wide variety of replies was chosen, suggesting that auditors use a wide
range of information in making a judgement. The most popular was evidence of
corrective actions, with 60 per cent of auditors replying. As this supplies direct
evidence of the quality loop and was identified in question 3 as an important
element in proving compliance, its popularity is to be expected. Thirty-seven per
cent of auditors considered an internal audit programme to be important
evidence. Quality procedures and records was the second most popular reply,
chosen by 44 per cent and documented evidence of reviews by 34 per cent, which
suggests that auditors prefer to rely on objective, verifiable evidence. However,
some auditors admitted to using more subjective evidence during an audit, such
as the 70 auditors who looked for staff understanding of their responsibilities and
the 66 who look for understanding of customer requirements.
Other
Understand customers
Processes monitored
Trained staff
Reaction to complaints
Staff understanding
Formal quality system
Audit programme
Quality records
Evidence of C A
Documented instructions
Staff committed
Job descriptions
Documented reviews
0
20
40
60
Number of replies
80
100
120
140
160
180
The use of objective evidence was supported by the answers to question 4b,
which asked Why do these types of evidence provide you with the information
necessary to judge whether a company complies with ISO 9001? This was an
open question, and about 60 per cent of the auditors mentioned the need for
objective evidence or proof in their answer.
Question 5a asked Who do you believe is the most important customer of
the registration process? This question attempted to find out who auditors
believed were supposed to benefit from registration. Four options were offered:
the audited company, the customers of the audited company, society at large
and others (see Figure 6). No one view predominated, with 48 per cent favouring
the audited company and 38 per cent preferring the customers of the audited
company (see Figure 6).
Question 5b asked the auditors to provide the reasons for their replies. This
was an open question so several categories were established and the replies
placed in the category that seemed closest to the auditors intent. The categories
were that the customer would receive assurance of the suppliers quality
Quality system
auditors
attitudes
45
Figure 5.
Question 4a, What do
you believe are the three
most important types of
evidence you can use in
establishing compliance
to ISO 9001?
IJQRM
13,8
Other
Society at large
46
Figure 6.
Question 5a, What do
you believe is the most
important customer of
the registration
process?
Customer of
audited company
Audited company
0
20
40
60
80
100
120
140
Number of replies
system, the customer would benefit from improvements in the suppliers quality
system, the supplier itself would benefit from the improvements, the supplier is
the customer of the certification body to which they were obligated or that
society would benefit. The first four answers received almost equal numbers of
replies (see Figure 7). This shows that auditors do not have a consensus view of
who are the customers of certification, and possibly the discussion about the
role of certification is mirrored in the auditors replies.
Other
Society benefits
Figure 7.
Question 5b, Why do
you believe they are the
most important
customer?
Direct customer of
certification body
Improved internal
operations
Improved supplier
benefits customer
Confidence in
supplier
0
10
20
30
40
50
60
Number of replies
The next two questions were concerned with problems that the auditors face
during an audit. Both questions were closed. Question 6a asked which two
factors proved to be the most common problems you have experienced during
the process of auditing a company? By far the most common replies were
companies possessing immature quality systems, with 74 per cent of auditors
mentioning this and companies not possessing fully documented procedures
with 46 per cent replying (see Figure 8). None of the other categories received
more than 30 replies.
50
100
150
200
250
Number of replies
Nearly half the auditors stated they had problems with companies not fully
documenting their procedures. Auditing a task requires a criterion against
which the task can be compared and the procedure usually provides the basis of
this criterion. If it is absent, the audit may be more difficult. Equally, the auditor
may decide that the company fails to comply and raise a non-conformance but
it is the authors observations from other research that auditors are unwilling to
raise a non-conformance without giving the company the opportunity to show
its compliance (which involves the auditor in more work).
Question 6a also listed several other difficulties auditors may have in
collecting evidence, such as unco-operative or unavailable staff. Very few
auditors have these problems.
Question 6b asked how often audits faced these problems. The pilot survey
suggested that auditors did not, in general, have many problems, so it was not
considered worthwhile asking how often each particular problem was
experienced. The full survey confirmed this result, with 78 per cent replying
never, once or twice or occasionally.
Question 7a asked While you have been personally auditing a company,
what two problems have you had in deciding whether a particular company
Quality system
auditors
attitudes
47
Figure 8.
Question 6a, Which
two factors have proved
to be the most common
problems you have
experienced during an
audit?
IJQRM
13,8
48
Figure 9.
Question 7a, What two
problems have you had
in deciding if a
company complies with
ISO 9001?
complies with the requirements of ISO 9001? A range of replies was received
(see Figure 9). The most common problem auditors faced was with companies
possessing overly complicated quality systems, with 45 per cent of the auditors
mentioning this. Auditors also suggested that companies often lack
understanding of quality and the certification process and that documented
procedures were not always followed. Several auditors mentioned that they
never had a difficulty in deciding compliance, as they believed that ISO 9000
demanded that the company maintains the evidence to ensure compliance, and
if there is a difficulty that is likely to mean that the company does not comply.
Question 7b asked how often these problems were faced, and again auditors
replied that they did not have many problems in deciding compliance, with 82
per cent replying never, once or twice or occasionally.
Other
No difficulty
Documented procedures not used
Complicated QMS
Low awareness of registration
Low awareness of quality
Other requirements
Auditing complex tasks
Auditing creative tasks
Identifying management commitment
Irrelevant clauses
0
20
40
60
80
100
120
140
Number of replies
Many replies in the other section in question 6a were replies in question 7a,
which suggested that the difference between questions 6 and 7 may not have
been sufficiently clear. The pilot survey suggested this problem and it was
hoped that fuller instructions might combat it, but unfortunately it was not
entirely effective. It is unlikely that the conclusions drawn have been
invalidated, as only a small proportion (6 per cent) of the answers were affected,
but the replies should be treated with some caution.
The survey concluded with several questions about the auditors
background and experience. Question 8 asked how long the auditor had been
with a certification body: 14 per cent had joined within the last year, 46 per cent
between one and three years ago, 26 per cent had joined between three and five
years ago and 14 per cent had joined more than five years ago. As ISO 9001
registration has increased rapidly in popularity in the last five years and
certification bodies have been recruiting to meet this demand it is not
surprising that many of them have only recently joined their certification body.
Question 9 asked whether they had any auditing experience before joining a
certification body. Experience was not clearly defined, but the purpose was to
discriminate between auditors who were familiar with the ideas of quality
system auditing before joining the certification body and those who were not.
The replies showed that 15 per cent had no previous experience, 34 per cent had
less than three years and 51 per cent had more than three years experience. It
appears that certification bodies try to recruit suitably experienced staff!
This is borne out by the answers to question 10, which asked what was the
auditors occupation before joining the certification body. Seventy-four per cent
had previous positions concerned with quality. Question 11 asked which sort of
company the auditors worked for before joining the certification body. The
categories were obtained from the Standard Industry Classification (SIC) codes.
The replies showed that auditors came from a wide background of companies
(see Figure 10). Most auditors come from manufacturing companies, possibly
because, at present, most certified companies are manufacturing companies
and are likely to have the most experience in operating a compliant quality
system (and so have experienced staff). Studies have shown that other
industries, such as software, are less experienced in applying quality
techniques[15].
Other
Consultancies
Construction
Utilities
Government/public
Retail/distributive
Food/drink
Consumer goods
Chemicals/process
Electrical/electronics
General manufacturing
Military/aerospace
Information technology
0
10
20
30
40
50
60
Number of replies
Additional results
The questionnaire also gave an opportunity to explore several other
hypotheses. One question was whether auditors are a homogeneous population
or whether the certification body they have joined influences the auditors
opinion, via training or experience. Initial examination of the results did not
highlight any major differences between the certification bodies in the replies to
the first seven questions. To test this, the results were examined using a Chisquare test (to the level of 0.005, all tests mentioned were to this level). This
showed that there was no statistically significant difference between the
auditors for the different certification bodies.
Although the certification bodies recruit auditors from a variety of
backgrounds, most of them have considerable previous experience in quality
Quality system
auditors
attitudes
49
Figure 10.
Question 11, What sort
of company were you
working for prior to
joining the certification
body?
IJQRM
13,8
50
Quality system
auditors
attitudes
51
IJQRM
13,8
52
8. Sayle, A., Management Audits: The Assessment of Quality Management Systems, 2nd ed.
McGraw-Hill, New York, NY, 1988.
9. Crosby, P., Quality Is Free, McGraw-Hill, New York, NY, 1979.
10. Deming, W. Edwards, Out of the Crisis, Cambridge University Press, Cambridge, 1986.
11. Perigord, M., Achieving Total Quality Management, Productivity Press, Cambridge, MA,
1987.
12. Oppenheim, A., Questionnaire Design, Interviewing and Attitude Measurement, 2nd ed.,
Pinter, London, 1992.
13. Weisberg, H. and Bowen, B., An Introduction to Survey Research and Data Analysis,
W.H. Freeman, San Francisco, CA, 1977.
14. DTI QA Register: The United Kingdom Register of Quality Assessed Companies, Vols 1
and 2, HMSO, London, 1994.
15. Davis, C.J., Gillies, A.C. ,Thompson, J.B. and Smith, P., Current practice in software quality
and the impact of certification schemes, Software Quality Journal, Vol. 2 No. 2, 1993,
pp. 145-61.