Software Review and Audit

Software Review and Inspection

Reviews provide a powerful way to improve quality by providing a means by which defects can be detected (and corrected) early in development.

Defects and Correctness

Defect:Anything that detracts from a programs ability to completely and effectively meet the users needs (Humphrey PSP) Correctness: A program is said to be correct only if it contains no defects. Correctness is only definable with respect to the users needs.

Verification and Validation

Verification and Validation

Validation: Are we building the right system? Verification: Are we building the system right?

Types of Review
There are a number of types of review ranging in formailty and effect. These include:
Buddy Checking
having a person other than the author informally review a piece of work. generally does not require collection of data difficult to put under managerial control generally does not involve the use of checklists to guide inspection and is therefore not repeatable.

Types of Review
Walkthroughs
generally involve the author of an artifact presenting that document or program to an audience of their peers The audience asks questions and makes comments on the artifact being presented in an attempt to identify defects often break down into arguments about an issue usually involve no prior preparation on behalf of the audience usually involve minimal documentation of the process and of the issues found process improvement and defect tracking are therefore not easy

Types of Review
Review by Circulation
similar in concept to a walkthrough artifact to be reviewed is circulated to a group of the author(s) peers for comment avoids potential arguments over issues, however it also avoids the benefits of discussion reviewer may be able to spend longer reviewing the artifact there is documentation of the issues found, enabling defect tracking usually minimal data collection

Types of Review
Inspection (Fagan 76)
formally structured and managed peer review processes involve a review team with clearly defined roles specific data is collected during inspections inspections have quantitative goals set reviewers check an artifact against an unambiguous set of inspection criteria for that type of artifact The required data collection promotes process improvement, and subsequent improvements in quality.

Software Inspection
The inspection process comprises three broad stages:
preparation collection Repair

Gilb and Graham [GilbGraham93] expand this three stage process into the inspection steps; Entry, Planning, Kickoff Meeting, Individual Checking, Logging Meeting, Root Cause Analysis Edit, Follow Up, Exit.

Benefits of Inspection
30% to 100% net productivity increases; Overall project time saving of 10% to 30%; 5 to 10 times reduction in test execution costs and time; Reduction in maintenance costs of up to one order of magnitude; Improvement in consequent product quality; Minimal defect correction backlash at systems integration time. In addition to these tangible benefits, less tangible benefits such as a training effect for inspectors are also evident.

Disadvantages
Up front costs (although far outweighed by benefit):
Training Implementation Support Ongoing allocation of staff resources

Not strictly repeatable

Typical Inspection Process

Entry
The author of the artifact requests that it be inspected. The artifact to be inspected is checked by the inspection moderator to ensure that certain entry criteria are met. The primary purpose of this stage is to ensure that inspection time is not wasted on artifacts that contain defects which the author should rightly have found.

Typical Inspection Process

Planning
The moderator determines the practical aspects of the inspection. This may include:
Determining the size and composition of the inspection team Determining the goals of the inspection. Determine the timing and purpose of the meetings.

Typical Inspection Process

Kickoff Meeting
Roles for the inspection team are assigned and clarified (generally the moderator does this). Documents, including the artifact, its source document (eg SRS for Design), the inspection checklist, and inspection rules are distributed and checked (Defects in the source document and checklist are sometimes found in these meetings, however, this meeting should be kept short). In some variations, the author(s) of the artifact may be required to give a quick walkthrough of the artifact to be inspected and its relation to the other documentation.

Typical Inspection Process

Individual Checking
The final stage of preparation is individual checking (although it could also be considered the main stage of collection). The majority of defects found in inspection processes are found in the individual checking stage. During this stage an individual reviewer reads the artifact and with the guidance of an inspection checklist attempt to find defects in the artifact. The reviewer should record any issues found. The reviewer should also make some effort to determine what they consider the severity of a defect to be and classify the defect.

Defect Severity & Classification

Severity
Major - Will cause problem if not corrected Minor - Will not If in doubt its major

Classification (for now)

Missing Extra Wrong

A Sample Form

Typical Inspection Process

Logging Meeting
A planned and moderated meeting with the primary purpose of logging the issues found by the reviewers. all reviewers should be given a chance to raise their issues as a scribe logs the issues being raised It is important that an issue is only logged once. moderator should ensure that discussion about issues is kept to a minimum in order to maintain the continuity of the meeting. Some variations of this process include group defect finding as an activity at the end of this meeting.

Typical Inspection Process

Edit
The editor (usually the author) is resposible for addressing all logged issues in the inspected artifact. The editor decides if something is a defect or not. All defects must be corrected. All non-defects should also be addressed in some way.

Typical Inspection Process

Follow Up and Exit
moderator checks that all defects have been addressed (and all non-defect issues addressed if required). moderator must also ensure that any defects found in a source document during inspection are forwarded to the owner of that document for correction. moderator may calculate certain metrics in this stage to be analysed to assess the effectiveness of an inspection. may also be used to hold a meeting to evaluate and recommend inspection process improvement An inspection will be exit when pre-defined set of inspection exit criteria have been satisfied.

Inspection Roles
Moderator / Leader Author / Producer Reviewer / Reader Scribe

Using the Log Form (1)

The first section describes the artifact being inspected, and summarises the individual reviewers data (from the review form in the previous stage if used).

Using the Log Form (2)

This section of the form contains information about major defects found and indicates which reviewers found them. Some variations also log minor defects here.

Using the Log Form (3)

The next section of the found simply totals the data in the columns for the reviewers.

Using the Log Form (4)

The final section of the formis used for metrics calculation.

Inspection Metrics

Total Defects Found = A + B - C, where A and B are the number found by reviewer A and B respectively and C is the number found by both A and B. Estimated Total Defects = AB/C Yield = Total Defecs Found / Estimated Total Defects * 100% Defect Density = Total Defects Found / Size Inspection rate = size / total inspection hours

Estimating Total Defects

Capture / Recapture Method
Capture a number of fish, tag them and release them (let this number be S1). Allow time for the first sample population to redistribute. Capture a second number of fish (let this number be S2). Count the number of tagged fish in the second population (let this number be ST). Calculate the proportion of tagged fish in the second population (let this number be T, then T=ST/S2). We assume that T is representative of the proportion of tagged fish in the total population (POP), so T*POP=S1, or for our purposes POP=S1/T.

I like fish but what about the defects

Let the number of defects found by one reviewer be the tagged population (A). Assume an even likelyhood of finding all defects (even distribution,...) Count the number of defects found by the second reviewer (B). Count the number of defects found by the second reviewer that were also found by the first (C the common defects). Calculate the proportion of common defects in the second reviewers defects (T=C/B). We assume that T is representative of the proportion of common defects in the total number of defects (EstTot), so T * EstTot=A, or for our purposes EstTot = A/T = (A*B)/C.

Calc. yield with > 2 reviewers

Sample Re-Inspection Criteria

Inspection rate too high Yield too low Majors found / Total found too low Unusual defect distribution High defect density

Should be specified in plan

AUDIT
The goal is to provide a guide to those responsible for software-related auditing and how best to achieve the final outcome of a fair, objective, and useful software-related audit that improves the situation as found. An independent examination of a work product or set of work products to assess compliance with specifications, standards, contractual agreements, or other criteria IEEE Purpose: to provide an independent evaluation of conformance of software products and processes to applicable regulations, standards, guidelines, plans, and procedures

Reasons
A specific project milestone has been reached and an audit is initiated as planned or as required by the auditing organizations charter. External parties or customers request an audit of a specific item, at a specific date, or at a project milestone. This could be part of a contract agreement. An internal organization has requested the audit, establishing a clear and specific need.

Software-related Audit
The client, person, or organization that requests the audit; The auditor or team who performs the audit; The auditee whose work is being examined.

Auditors Responsibilities
Determining the team size; Briefing team members on the audit scope and areas to be audited; Providing background about the organization being audited; Assigning the workload of who will audit what areas; Determining the audit schedule; Notifying and briefing the audited organization on the scope of the audit and materials that need to be provided; Ensuring that the audit team is prepared to conduct the audit; Ensuring that the audit plan or procedures are performed; Issuing reports in accordance with the audit plan or procedures.

Auditees Responsibilities
Establishing a professional, positive attitude about the audit among the members of the audited organization; Participating in the audit; Providing all relevant materials and resources to the audit team; Understanding the concerns of the auditors and verifying their factual accuracy; Providing a response to the audit report; Correcting or resolving deficiencies cited by the audit team.

Types of Software Audits

Software piracy audit Security audit Information systems audit ISO 9001:2000 software audit CMMI-DEV appraisal Personal audit experiences Automated audits

Software Piracy Audit

Security Audit
Issues
Backups Antivirus. Firewall Access control.

Security Audit
ISO 17799 It is organized into 10 sections
Business continuity planning; Systems access control; System development and maintenance; Physical and environmental security; Compliance; Personnel security; Security organization; Computer and operations management; Asset classification and control; Security policy.

Security Audit

Security Audit

Information Systems Audit

Information systems auditing evaluates whether computer-based information systems safeguard assets, maintain data integrity, achieve organizational objectives effectively, and consume resources efficiently. Ron Weber

Information Systems Audit

The information systems lead auditor must be sensitive to the following
All audits should be conducted only with prior approval of the management. Consult an advocate for all the applicable legislationthat is, if you do not want to be caught by surprise later. Ensure that one does not violate the software copyright in any form. Ask the accounts department how long they retain financial records. Ensure that there is no misuse of the information processing facilities by any person, insider as well as outsider. If you are traveling with your notebook PC where you have stored encrypted files, you may be breaking a few laws of the land.

Information Systems Audit

Information Systems Audit

Information Systems Audit

The information systems lead auditor must be sensitive to the following
All audits should be conducted only with prior approval of the management. Consult an advocate for all the applicable legislationthat is, if you do not want to be caught by surprise later. Ensure that one does not violate the software copyright in any form. Ask the accounts department how long they retain financial records. Ensure that there is no misuse of the information processing facilities by any person, insider as well as outsider. If you are traveling with your notebook PC where you have stored encrypted files, you may be breaking a few laws of the land.

ISO 9001:2000 Software Audit

ISO 9001:2000 Software Audit

ISO 9001:2000 Software Audit

ISO/IEC 90003:2004 explains how ISO 9001:2000 can be applied to software related services. Sample Checklist
Provide quality infrastructure
Identify infrastructure needs. Provide needed infrastructure. Maintain your infrastructure. Maintain the tools you need in order to manage software.

Control software design and development

Plan software design and development Plan software design and development.

ISO 9001:2000 Software Audit

Identify infrastructure needs.
Identify the infrastructure you need in order to develop software. Identify the hardware you need in order to develop software. Identify the software you need in order to develop software. Identify the facilities you need in order to develop software. Identify the tools you need in order to manage software. Identify the tools you need in order to develop software. Identify the tools you need in order to support software. Identify the tools you need in order to protect software. Identify the tools you need in order to control software.

CMMI-DEV appraisal

CMMI-DEV appraisal
Phase 1 1.1 1.2 1.3 1.4 Process Plan and Prepare for Appraisal Analyze Requirements Develop Appraisal Plan Select and Prepare Team Obtain and Inventory Initial Objective Evidence

1.5
2 2.1 2.2 2.3 2.4 2.5 2.6 3 3.1 3.2

Prepare for Appraisal Conduct

Conduct Appraisal Prepare Participants Examine Objective Evidence Document Objective Evidence Verify Objective Evidence Validate Preliminary Findings Generate Appraisal Results Report Results Deliver Appraisal Results Package and Archive Appraisal Assets

CMMI-DEV appraisal
Benefits to the organization:
Improved accuracy in appraisal results delivered by external appraisal teams (i.e., clear understanding of implemented processes, strengths, and weaknesses); Detailed understanding of how each project or support group has implemented CMMImodel practices, and the degree of compliance and tailoring of organizational standard processes; Assets and resources for monitoring process compliance and process improvement progress; Residual appraisal assets that can be reused on subsequent appraisals, minimizing the effort necessary for preparation.

Personal Audit Experiences

Automated Audits