Software Process Reviews/Audits

Process Overview
by
Tom Gilchrist, CSQA, CSQE,
 Before we start…

Information in this presentation are

my opinions and not necessary
those of my employer.

• SQA Context
• Overview of SW Audit Process
• SW Audit Examples

SASQAG 10/17/2002
tomg@tomgtomg.com 2
 Some Terms/Ideas

• Process
• Deterministic vs. Non
Deterministic
• Quality vs. Value

SASQAG 10/17/2002
tomg@tomgtomg.com 3
 Software Quality Assurance

• Check software products and processes to verify that they

comply with the applicable procedures and standards.
(Process Reviews or Audits)
• Review and measure the quality of software products and
processes throughout development. (Dynamic & Static
Testing)
• Provide software project management (and other appropriate
parties) with the results of reviews and process checks.
• Work with the software project during early stages to
establish plans, standards, and procedures to keep errors
from occurring in the first place.

SASQAG 10/17/2002
tomg@tomgtomg.com 4
 Formal Definition

Audits provide an independent evaluation of

software products or processes to ascertain
compliance to standards, specifications, and
procedures based on objective criteria that included
documents that specify:
– The form or content of the product to be
produced
– The process by which the products shall be
produced
– How compliance to standards or guidelines
shall be measured.
IEEE STD 1028, (1988)

SASQAG 10/17/2002
tomg@tomgtomg.com 5
 Audit Types

• First Party Audit

– Within you company or organization
• Second Party Audit
– Sometimes called “external audits”
– By a Customer on his Supplier
– By a Supplier on you.
• Third Party Audit
– Outside third party is contracted to do
the audit.
SASQAG 10/17/2002
tomg@tomgtomg.com 6
 Audit/Process Review Principles

• Conducted by individuals who are

organizationally independent of the developers.
• Begin early in the requirements phase and
continue throughout the development process.
• Professionally planned, conducted and
documented.
• Follow-up on corrective action.
• Project Management is involved in the Audit
process and is responsible for rework and
process improvements.

SASQAG 10/17/2002
tomg@tomgtomg.com 7
 What Software Audit Should Do

• Determine:
• Compliance to requirements
• Conformance to plans, policies, procedures, and
standards
• Drive process improvement based on:
• Adequacy of plans, policies, procedures, and
standards
• Effectiveness and efficiency of plans, policies,
procedures, and standards
• Assess personnel familiarity to requirements and
documentation
• Assure availability, use and adherence to software
standards
SASQAG 10/17/2002
tomg@tomgtomg.com 8
 What Triggers an Audit?

• Quality Assurance Plan

• Event
• Date
• Requests from management
• Requests from developers
• Requests from customers
• Integration with process improvement activities
• Outside requirements — regulatory
• Gut feel

SASQAG 10/17/2002
tomg@tomgtomg.com 9
 Scope: Requirements, Time, and Target

External
Standards

• Spread around
organization
Audit
Target
• Cover all functions and
activities
• Try to hit things early
Organizational
Procedures and • Move towards process
Methods
audits

SASQAG 10/17/2002
tomg@tomgtomg.com 10
 Process Review/Audit Process

Developers Auditor Project Manager

Plan
Prepare
Start (Requirements,
Audit
Scope, & Checklist)

Conduct Write-up Review

Audit Report & with NO
Findings Manager

Findings?

Corrective YES
Actions

OK Closeout
Audit &
File END

Follow-up
Audit
Re-Work

SASQAG 10/17/2002
tomg@tomgtomg.com 11
 Identify Requirements

• Policies/Standards Corporate, Group, IEEE

• Processes/Plans SCMP, SQAP, SDP, Project Plan
• Procedures Change Management, Design
Reviews, Document Standards,
Testing
• Task Instructions Library updates, unit testing, peer
reviews

• Success of an audit is directly proportional to preparation,

research and analysis conducted before the audit is
performed.

SASQAG 10/17/2002
tomg@tomgtomg.com 12
 Requirement Types

• Functional (ascertainably true or false)

• Quality (range of acceptable values)

SASQAG 10/17/2002
tomg@tomgtomg.com 13
 Types of Audits (Internal)

• Quality System Audits

• Product Audit
• Process Audit
• Project Audit
• CM Audit

SASQAG 10/17/2002
tomg@tomgtomg.com 14
 Evidence Collection

• Collect Factual Information

• Analyze and Evaluate the Evidence
• Draw Conclusions
• Generate Findings

SASQAG 10/17/2002
tomg@tomgtomg.com 15
 Corrective Action of Findings

• Determine Action
– Immediate Remedial Action
– Process Improvement/Fix
– Acceptable Risk
• Identify Root Cause
• Corrective Actions Plan
• Manage CA Plan to completion
• Analyze Effects of CA

SASQAG 10/17/2002
tomg@tomgtomg.com 16
 Develop Audit Checklist

• Focus on clear requirements (or

unclear to fix)
• Select subset of requirements
• Focus on important steps/products
• Write clear concise questions
• Canned checklist vs. straw horse

SASQAG 10/17/2002
tomg@tomgtomg.com 17
 Checklist Sample
Requirement Checklist Item Details Observations Results (P/F)

 
 
Company Does project QA plan Check SQA document for a list    
Standard ABC- will have a list of of approved peer reviews and
234, page 7 deliverables subject to which documents are to be
Peer Reviews? reviewed. (if no documents are
found, then fail. If no peer
review procedures are
referenced, then fail)

Project SQA Were the number of Check to see which audits were    
Plan audits completed planned for the last 60 days.
equal to the number Check for evidence that the audit
planned? was completed and if there were
findings, that a CA plan was
signed.

Project SQA Were the number of For each peer review type, check    
Plan peer reviews the CM records for the past 60
  completed equal to the days to see if the document type
number planned? specified in the QA plan was
checked into CM for the first
time. If so, check for records of
the peer review being completed
as per peer review process cited
in SQA plan.

SASQAG 10/17/2002
tomg@tomgtomg.com 18
 Interviewing

• Ask open-ended questions

• Know the types of answers expected
• Focus on Process and not People
• Seek Corroboration and Evidence

SASQAG 10/17/2002
tomg@tomgtomg.com 19
 Sample Interview Questions

• How do you track your progress?

• Do you have a CM Plan?
• Tracing
– What are you working on?
– Is it a configured item?
– Do you have an approved CR or PR?
– Is the version you are working on
checked out of CM?

SASQAG 10/17/2002
tomg@tomgtomg.com 20
 Desirable Auditor Characteristics
• Emotional • Mechanical
• Interviews • Sampling
• Group • Root Cause
dynamics Analysis
• Oral reports • Intellectual
• Empathy • Writing
• Don’t take • Planning
things • Speaking
personally • Detail
Oriented
• Concise
SASQAG 10/17/2002
tomg@tomgtomg.com 21
 Desirable Auditor Characteristics
(Cont.)

• Knowledge of Audit process

• Knowledge of target (SW) processes
• Knowledge of techniques
• Professional attitude
• Good listener
• Inquisitive/analytical
• Communicates at all levels
• Detailed Notes and Observations
• Diplomatic

SASQAG 10/17/2002
tomg@tomgtomg.com 22