LECTURE 8: Software Quality Assurance

Software Quality Assurance

For example, if you are developing the next Flash video game, are
you sure your code will run in every browser, with any recent version
of Flash, on all platforms, all while making sure that your game still
performs adequately?  Most people can’t get this kind of coverage
cost effectively.

2
 Quality Assurance

Software quality
attributes

 Reliability
 Functionality Product in operation
 Usability
(user perspective)
 Efficiency

•Maintainability
•Reusability Product revision
•Portability (developer perspective)
•Testability 3
 Reliability
 Probability the system operates without failure in a given period of
time under a given operational environment

 But what exactly is a Software failure?

Formal view: deviation from specified behaviour

Engineering view: deviation from required, specified or expected
behaviour

4
5
 Difference between error, fault, bug, failure and defect
What is an error?
Error is deviation from actual and expected value. It represents mistake made by people.

What is a fault?
Fault is incorrect step, process or data definition in a computer program which causes the
program to behave in an unintended or unanticipated manner.
It is the result of the error.

What is a bug?
Bug is a fault in the program which causes the program to behave in an unintended or
unanticipated manner.
It is an evidence of fault in the program.

What is a failure?
Failure is the inability of a system or a component to perform its required functions within
specified performance requirements.
Failure occurs when fault executes.

What is a defect?
A defect is an error in coding or logic that causes a program to malfunction or to produce
incorrect/unexpected results.
A defect is said to be detected when a failure is observed.

http://softwaretestingbykunti.blogspot.fr/2012/11/difference-between-error-fault-bug.htm
l
 Relationship between faults and failures

1. Most faults are benign

2. For most faults: removal will not lead to greatly improved reliability
3. Large reliability improvements only come when we eliminate the small proportion of
faults which lead to the more frequent failures
4. Does not mean we should stop looking for faults, but warns us to be careful about
equating fault counts with reliability 7
Using fault data to predict reliability
Pre-release testing Post-release operation

delivery
Pre-release vs post-release faults?

Actual

9
 The problem with ‘problems’
 Defects
 Faults
 Failures
 Bugs
 Anomalies
 Crashes

Incident Types
Failure (in pre or post release)
Fault
Change request

10
 Recording incidents
Applicable to all incident types
What: Product details
Where (Location): Where is it?
Who: Who found it?
When (Timing): When did it occur?
What happened (End Result): What was observed?
How (Trigger): How did it arise?
Why (Cause): Why did it occur?
Severity/Criticality/Urgency
Change
Example: Failure Data
What: ABC Software Version 2.3
Where: Adel’s home PC
Who: Adel
When: 13 Jan 2020 at 21:08 after 35 minutes of operational use
End result: Program crashed with error message xyz
How: Loaded external file and clicked the command Z.
Why: <BLANK - refer to fault>
Severity: Major
Change: <BLANK> 11
 Example: Fault Data (1) - responsive
What: ABC Software Version 2.3
Where: Function <abcd> in Module <ts0023>
Who: Salah
When: 14 Apr 2017, after 2 hours investigation
What happened: Caused reported failure id <0096>
How: <BLANK>
Why: Missing exception code for command Z
Urgency: Major
Change: exception code for command Z added to function <abcd> and also to
function <efgh>. Closed on 15 Jan 2007.

Example: Fault Data (2) - reactive

What: ABC Software Version 2.3
Where: Help file, section 5.7
Who: Adel
When: 25 Nov 2019, during formal inspection
End result: Likely to cause users to enter invalid passwords
How: The text wrongly says that passwords are case sensitive
Why: <BLANK>
Urgency: Minor
Change: Suggest rewording as follows ... 12
 Example: Change Request
What: ABC Software Version 2.3
Where: File save menu options
Who: Adel
When: 02 Jul 2018
End result: <BLANK>
How: <BLANK>
Why: Must be able to save files in ascii format - currently not possible
Urgency : Major
Change: Add function to enable ascii format file saving

Tracking incidents to components

Incidents need to be traceable to identifiable components - but at what
level of granularity?
 Unit
 Module
 Subsystem
 System
13
 Problem Report
Product name
Version
Component
Hardware platform
Operating System
Problem summary
Details
Reproducibility
Actual Results
Expected Results
Severity
Additional information

14
The Software Engineering Institute (SEI)
Capability Maturity Model

15
 Quality vs. CMM Level

Consider 500K SLOC

1000
Defect rate
900
halved per
800
maturity level Level 2 450 defects, rework
In-Process Defects/MAELOC

700
equals 16 hrs/defect
600
estimated at $100/hr
$1,600 x 450 =
500

400
$720K rework
300

200 Level 3 $360K rework

Level 4 $180K rework
100

0
1 2 3 4 5
Software CMM Level
*MAELOC = million assembly-equivalent lines of code
Level 5 $90K rework
• A defect is a bug or error that escapes the phase in which it was
introduced
• M.SLOC
Diaz and J. = software
Sligo, linesImprovement
“How Software Process of code Helped Motorola,” IEEE Software,September/October 1997, p. 76.
16
 History of Inspections

“...a formal evaluation technique in which software requirements, design or

code are examined in detail by a person or group other than the author to
detect faults, violations of development standards and other problems”
IEEE Standard for Software Reviews and Audits
 Original method developed by Fagan at IBM
 In use since early 1970s
 Highly formalised method
 Reliant on generation of quantitative defect data

17
 Motivation for Inspections: Cost to Fix a Fault

Cost to fix a fault by development phase

The Inspection Process

18
 The Inspection Process
Inspection Stage Description
Planning Identifies work product to be inspected and sets the inspection schedule.
Overview Optional phase where team members who are unfamiliar with the work
product to be inspected receive orientation.
Preparation Team members inspect the work individually looking for defects in the
work product.
Inspection Meeting Inspection team members met to discuss possible defects in the work
product.
Rework The work product is revised to conform to requirements and
specifications.
Follow up The rework is verified, final inspection data is collected and summarized,
and the inspection is officially closed.

19
 Inspections: Defect Logging Meeting
“Defect Logging” meeting: - Record defects from individual checking; -
Group Checking; - Record process improvement suggestions

Requirements Inspect System Test

Architecture Integration Test

Design Functional Test

Code Unit Test

Product 20
 Inspections: Management Issues
 Only the quality of the product should be considered when evaluating an
individual’s work

 Goal for inspections is to detect as many defects as possible

 Remember the number of defects will depend on the problem’s complexity

and the skills of staff and on thoroughness of inspection

 Don’t use defects found as a measure of individual performance

 Do not punish individuals for defects

Inspections: Critical Success Factors
Logging meeting must focus on defect detection not discussion

Logging meeting should strive to create synergy (collaboration)

Optimum checking rates should be respected (approx. 1 page per hour)

Data collection should be structured and automated for later analysis 21

 Inspections: Costs and Benefits
 Benefits
– claimed productivity increases of up to 100%
– claimed 10 fold reduction in testing costs
– 2/3 reduction in maintenance costs
– team more likely to deliver system on time

 Costs
– consume up to 15% of development budget
– high training costs (1 week per person)
– data analysis effort
– meetings perceived as overhead

Inspections Vs Testing
Not mutually exclusive alternatives
Testing done on executable code
Inspection performed on documents or source code
Important to inspect test cases and documentation before testing is
performed 22
 References

 Bernd Bruegge & Allen H. Dutoit, Object-Oriented Software

Engineering: Using UML, Patterns, and Java
 Software Engineering, Ivan Marsic, 2020
 Sommerville, I. (2015). Software Engineering 10. Pearson.