QA 003

SOFTWARE QUALITY ASSURANCE

TIP Vision
As a CHED Center of Excellence or Center of Development in at least three Engineering programs; As a school steeped in research and community service; As a school that is managed efficiently with computerized operations capable of having fast access to accurate information in order to respond to an ever-changing environment to ensure financial viability; and As a school with: Level III Accredited Status for 75 percent of its programs Total Quality Management for all aspects of operations including ISO Certifications in the provision of all its academic offerings The best work environment for its teaching and non-teaching employees, and Strategic alliances with other schools and industry

TIP Mission
The Technological Institute of the Philippines is committed to:

bring the blessings of higher education within reach of Filipinos; to maintain the highest standard of instruction and to constantly redefine the meaning of academic life, and to transform students into graduates with full competence in their fields of study and who also possess: Filipino values. The Filipino values of honesty and integrity, service to others, the importance of family, frugality, resilience in the face of adversity, and the willingness to surmount difficulties in order to succeed and excel. Industry-desired values. The industry-desired values of positive work attitude, good communication skills, proficiency in computers and in the software that pertain to their fields of study, and the openness to keep on learning to reinvent themselves. Global citizen values. The global values of respect for cultural diversity, care for the environment and the desire to contribute to the general welfare of society.

Grading System
PG = 0.50PE + 0.50 CSP MG = 1/3PG + 2/3 ( 0.50ME + 0.50CSM) FG = 1/3MG + 2/3( 0.50FE + 0.50CSF)

Course Objectives
This course aims to :

1. Provide students with the knowledge and understanding about the characteristics of a good software. 2. Equip students with the skills on how to assess and evaluate software with good design and codes. 3. Familiarize students with the software components that can be considered as basis for evaluation.

Intended Course Outcomes

At the end of the course, the students should be able to: 1. Examine the vital components of a good quality software. 2. Evaluate the components of a good quality application. 3. Compare poor software design from a better one. 4. Recommend better software design. 5. Identify ways of maintaining quality software. 6. Reflect on personal transformation along the TIP.

Course Policies
Course Requirements: Attendance (at least 80% of total class hours)
o Unannounced quizzes o Graded Recitations

Major Examinations and Quizzes Research Works

o Miniskirt rule

Case Studies

Class Policies
English as medium Explained and Unexplained absences No special anything
exams quizzes projects

One shot policy on HONESTY Transparency

Course Coverage
Software Quality Definition of software Software errors, faults and failures Classification of the causes of software errors Software quality definition Software quality assurance definition and objectives Software quality assurance and software engineering

Course Coverage
Software Quality Factors The need for comprehensive software quality requirements Classifications of software requirements into software quality factors Product operation software quality factors Product revision software quality factors Product transition software quality factors

Course Coverage
Components of SQA System: An overview The SQA system an SQA architecture Pre-project components Software project life cycle components Infrastructure components for error prevention and improvement Management SQA components SQA standards, system certification, assessment components

Course Coverage
Pre-project software quality components Contract review Introduction: the CFV Project completion celebration The contract review process and its stages Contract review objectives Implementation of a contract review Contract reviews for internal projects

Course Coverage
Development and Quality plans Development plan and quality plan objectives Elements of the development plan Development and quality plans for small projects and for internal projects

Course Coverage
Integrating quality activities into the project life cycle Classic and other software development methodologies Factors affecting intensity of quality assurance activities in the development process Verification, validation and qualification A model for SQA defect removal effectiveness and cost

Course Coverage
Reviews Review objectives Formal design reviews (DRs) Peer reviews A comparison of the team review methods Expert opinions

Course Coverage
Software testing - Definition and objectives - Software testing strategies - Software test classifications White box testing Black box testing.

Course Coverage
Software testing implementation The testing process Test case design Automated testing Alpha and beta site testing programs

Course Coverage
Assuring the quality of software maintenance components Assuring the quality of external participants contributions CASE tools and their effect on software quality What is a CASE tool? The contribution of CASE tools to software product quality The contribution of CASE tools to software maintenance quality The contribution of CASE tools to improve project management

Course Coverage
Software quality infrastructure components Procedures and work instructions 1. The need for procedures and work instructions 2. Procedure and procedure manuals 3. Work instruction and work instruction manuals 4. Procedures and work instruction: preparation, implementation and updating

Course Coverage
Supporting quality devices

Templates Checklist

Course Coverage
Staff training and certification 1. Introduction: Surprises for the 3S development team 2. The objectives of training and certification 3. The training and certification process 4. Determining professional knowledge requirements 5. Determining training and updating needs 6. Planning training and updating programs 7. Defining positions requiring certification 8. Planning the certification processes 9. Delivery of training and certification programs 10. Follow-up subsequent to training and certification

Software

Software
is a general term used to describe a collection of computer programs and procedures and documentation that perform some tasks on a computer system includes application software such as word processors which perform productive tasks for users, system software such as operating systems, which interface with hardware to provide the necessary services for application software, and middleware which controls and co-ordinates distributed systems sometimes used in a broader context to mean anything which is not hardware but which is used with hardware, such as film, tapes and records.

Software
Computer programs, procedures, and possibly associated documentation and data pertaining to the operation of a computer system. [IEEE_Std_610.12-1990]

Nature of Software
Invisibility Intangibility Limited opportunities to detect defects (bugs) New and demanding functionality Extraordinarily high complexity

SW Devt Environment
Being contracted Subjection to customer-supplier relationship Requirement for teamwork Need for cooperation and coordination with other development teams Need for interfaces with other software systems Need to continue carrying out a project while the team changes Need to continue maintaining the software system for years

SW Devt Environment

Causes of software errors

1. faulty requirements definition 2. client-developer communication failures 3. deliberate deviations from software requirements 4. logical design errors

5. coding errors
6. non-compliance with documentation and coding instructions

7. shortcomings of the testing process

8. procedure errors 9. documentation errors

Errors / Faults / Failures

"software error" "software fault" "software failure" types of errors 1. code error 2. procedure error 3. documentation error 4. software data error

What is the difference between error, fault, and failure?

Error
a derivation of the required operation of the system or subsystem. a human action that produces an incorrect result

Fault
A defect in the system an incorrect step, process, or data definition in a computer program an underlying condition within a software that causes certain failures to occur

Failure
occurs when the system fails to perform its required function the inability of a system or component to perform its required functions within specified performance requirements behavioral deviation from the user requirement or product specification

Fault / Failure Chain

Fault >> error
a fault which has not been activated by the computation process is dormant a fault is active when it produces an error

Error >> failure

an error is latent when it has not been recognized an error is detected by a detection algorithm/mechanism

Failure >> fault

a failure occurs when an error passes through and affects the service delivered a failure results in a fault for the system which contains or interacts with

Fault Avoidance
Activities: Preventing the introduction or occurrence of faults (fault prevention): SWT, formal methods, high level languages, CASE tools, Reducing the number and seriousness of faults (fault removal): Analysis diagnosis correction Evaluating the present number future incidents, and severity of faults (fault forecasting): Statistics & management

Famous Software Errors

Therac-25 Patriot Missile System NASA's Mars Polar Lander ESA's Ariane 5 Launch System 2003 Blackout

Therac-25
When operating in soft X-ray mode, the machine was designed to rotate three components into the path of the electron beam, in order to shape and moderate the power of the beam. The accidents occurred when the high-energy electron-beam was activated without the target having been rotated into place; the machine's software did not detect that this had occurred, and did not therefore determine that the patient was receiving a potentially lethal dose of radiation, or prevent this from occurring. The design did not have any hardware interlocks to prevent the electronbeam from operating in its high-energy mode without the target in place.

Therac-25
The engineer had reused software from older models. These models had hardware interlocks and were therefore not as vulnerable to the software defects. The hardware provided no way for the software to verify that sensors were working correctly. The equipment control task did not properly synchronize with the operator interface task, so that race conditions occurred if the operator changed the setup too quickly. This was evidently missed during testing, since it took some practice before operators were able to work quickly enough for the problem to occur. The software set a flag variable by incrementing it. Occasionally an arithmetic overflow occurred, causing the software to bypass safety checks.

Patriot Missile System

On February 25, 1991, the Patriot missile battery at Dharan, Saudi Arabia had been in operation for 100 hours, by which time the system's internal clock had drifted by one third of a second. For a target moving as fast as an inbound TBM, this was equivalent to a position error of 600 meters. The radar system had successfully detected the Scud and predicted where to look for it next, but because of the time error, looked in the wrong part of the sky and found no missile. With no missile, the initial detection was assumed to be a spurious track and the missile was removed from the system. No interception was attempted, and the missile impacted on a barracks killing 28 soldiers.

Mars Polar Lander

The last telemetry from Mars Polar Lander was sent just prior to atmospheric entry on December 3, 1999. No further signals have been received from the lander. The cause of this loss of communication is unknown. According to the investigation that followed, the most likely cause of the failure of the mission was a software error that mistakenly identified the vibration caused by the deployment of the lander's legs as being caused by the vehicle touching down on the Martian surface, resulting in the vehicle's descent engines being cut off whilst it was still 40 meters above the surface, rather than on touchdown as planned. Another possible reason for failure was inadequate preheating of catalysis beds for the pulsing rocket thrusters

Ariane 5 Rocket
June 4, 1996 was the first test flight of the Ariane 5 launch system. The rocket tore itself apart 37 seconds after launch, making the fault one of the most expensive computer bugs in history. The Ariane 5 software reused the specifications from the Ariane 4, but the Ariane 5's flight path was considerably different and beyond the range for which the reused code had been designed. Specifically, the Ariane 5's greater acceleration caused the back-up and primary inertial guidance computers to crash, after which the launcher's nozzles were directed by spurious data. Pre-flight tests had never been performed on the realignment code under simulated Ariane 5 flight conditions, so the error was not discovered before launch. Because of the different flight path, a data conversion from a 64-bit floating point to 16-bit signed integer caused a hardware exception (more specifically, an arithmetic overflow, as the floating point number had a value too large to be represented by a 16-bit signed integer). Efficiency considerations had led to the disabling of the exception handler for this error. This led to a cascade of problems, culminating in destruction of the entire flight.

2003 North America Blackout

August 14, 2003
12:15 p.m. Inaccurate data input renders a system monitoring tool in Ohio ineffective. 1:31 p.m. The Eastlake, Ohio, generating plant shuts down. 2:02 p.m. First 345-kV line in Ohio fails due to contact with a tree in Walton Hills, Ohio. 2:14 p.m. An alarm system fails at FirstEnergy's control room and is not repaired. 2:27 p.m. Second 345-kV line fails due to tree. 3:05 p.m. A 345-kV transmission line fails in Parma, south of Cleveland due to a tree. 3:17 p.m. Voltage dips temporarily on the Ohio portion of the grid. Controllers take no action, but power shifted by the first failure onto another 345-kV power line causes it to sag into a tree. While Mid West ISO and FirstEnergy controllers try to understand the failures, they fail to inform system controllers in nearby states. 3:39 p.m. A First Energy 138-kV line fails. 3:41 and 3:46 p.m. Two breakers connecting FirstEnergys grid with American Electric Power are tripped as a 345-kV power line and 15 138-kV lines fail in northern Ohio. Later analysis suggests that this could have been the last possible chance to save the grid if controllers had cut off power to Cleveland at this time. 4:06 p.m. A sustained power surge on some Ohio lines begins uncontrollable cascade after another 345-kV line fails. 4:09:02 p.m. Voltage sags deeply as Ohio draws 2 GW of power from Michigan. 4:10:34 p.m. Many transmission lines trip out, first in Michigan and then in Ohio, blocking the eastward flow of power. Generators go down, creating a huge power deficit. In seconds, power surges out of the East, tripping East coast generators to protect them, and the blackout is on. 4:10:37 p.m. Eastern Michigan grid disconnects from western part of state. 4:10:38 p.m. Cleveland separates from Pennsylvania grid. 4:10:39 p.m. 3.7 GW power flow from East through Ontario to southern Michigan and northern Ohio, more than ten times larger than the condition 30 seconds earlier, causing voltage drop across system. 4:10:40 p.m. Flow flips to 2 GW eastward from Michigan through Ontario, then flip westward again in a half second. 4:10:43 p.m. International connections begin failing. 4:10:45 p.m. Western Ontario separates from east when power line north of Lake Superior disconnects. First Ontario plants go offline in response to unstable system. Quebec is protected because its lines are DC, not AC. 4:10:46 p.m. New York separates from New England grid. 4:10:50 p.m. Ontario separates from Western New York grid. 4:11:57 p.m. Last lines between Michigan and Ontario fail.

4:13 p.m. End of cascade. 256 power plants are off-line. 85% went offline after the grid separations occurred, most of them on automatic controls. 50 million people without power.

Cost of Errors
"Software bugs, or errors, are so prevalent and so detrimental that they cost the U.S. economy an estimated $59.5 billion annually, or about 0.6 percent of the gross domestic product. Although all errors cannot be removed, more than a third of these costs, or an estimated $22.2 billion, could be eliminated by an improved testing infrastructure that enables earlier and more effective identification and removal of software defects. These are the savings associated with finding an increased percentage (but not 100 percent) of errors closer to the development stages in which they are introduced. Currently, over half of all errors are not found until "downstream" in the development process or during post-sale software use." US Dept of Commerce June 2002

Quality
Simplistically, means that a product should meet its specification. This is problematical for software systems There is a tension between customer quality requirements (efficiency, reliability, etc.) and developer quality requirements (maintainability, reusability, etc.); Some quality requirements are difficult to specify in an unambiguous way; Software specifications are usually incomplete and often inconsistent.

Approaches to Quality
Transcendental view: universally identifiable, absolute, unique and perfect Product view: measurable in an objective manner User view: fitness for use Manufacturing view: the result of the right product development Value-based view (Economic): a function of costs and benefit

What is Software Quality?

Software Quality
Conformance to explicitly stated functional and performance requirements, explicitly documented development standards, and implicit characteristics that are expected of all professionally developed software. (Pressman)

Software Quality
1. The degree to which a system, component, or process meets specified requirements. 2. The degree to which a system, component, or process meets customer or user needs or expectations. (IEEE)

Software Quality
Process-based Quality
There is a straightforward link between process and product in manufactured goods. More complex for software because:
The application of individual skills and experience is particularly important in software development; External factors such as the novelty of an application or the need for an accelerated development schedule may impair product quality.

Care must be taken not to impose inappropriate process standards - these could reduce rather than improve the product quality.

Software Quality
Quality the degree of excellence of something. We measure the excellence of software via a set of attributes. ( satisfying requirements)!
- Quality is absolute or at least independent of the requirements underlying the product. - It is part of the software development to get the right requirements.

( user satisfaction)!
McDonalds restaurants are popular, but

Classical Quality Assurance (Hardware)

Requirements: complete set of external quality characteristics --> complete set of internal quality characteristics and a detailed and complete set of requirements and specifications Design:
design that satisfies the requirements and specifications design for reliability (wear out) design for manufacturability design for maintainability (e.g., self-diagnosis)

Classical Quality Assurance (Hardware)

Manufacturing: statistical production process control with acceptance sampling Operation: collect failure data for continuous improvement and predictions (intelligent maintenance)

SW Quality Assurance
1. A planned and systematic pattern of all actions necessary to provide adequate confidence that an item or product conforms to established technical requirements. 2. A set of activities designed to evaluate the process by which the products are developed or manufactured. Contrast with quality control.

- IEEE

Software Quality Assurance

Software quality assurance is the systematic, planned set of actions necessary to provide adequate confidence that a software development or maintenance process conforms to established functional technical requirements as well as the managerial requirements of keeping to schedules and operating within the budget

Software Quality Assurance

the entire software development processmonitoring and improving the process, making sure that any agreed-upon standards and procedures are followed, and ensuring that problems are found and dealt with. It is oriented to 'prevention'

SW Quality Assurance
Requirements: Completeness is hard to achieve (complexity) Design: design for reliability only in special cases design for manufacturability is not required design for maintainability in special cases Focal area of software quality assurance! Manufacturing >> Implementation: limited success with statistical process control (metrics) Operation: Fixing found bugs

Control vs. Assurance

Quality control is a set of activities carried out with the main objective of withholding products from shipment if they do not qualify. Quality assurance is meant to minimize the costs of quality by introducing a variety of activities throughout the development and maintenance process in order to prevent the causes of errors, detect them, and correct them in the early stages of development.

SW ENGINEERING
The application of systematic, disciplined, quantifiable approach to the development, operation and maintenance of software; that is, the application of engineering to software

The Big Question

Q: How do we assure quality?
A: We need to have a good process.

QUALITY MODELS
Such general definitions of software quality are not sufficient in practice Thus, software quality is described by specific quality models causal relationship from intangible quality views to tangible software measures

Software Quality Factors

QUALITY FACTORS
A quality factor represents a behavioral characteristic of the system.
Operation Revision Transition

A quality criterion is an attribute of a quality factor that is related to software production and design. A quality metric is a measure that captures some aspect of a quality criterion.

ISO / IEC 9126

ISO / IEC 9126

1. Functionality

2. Reliability
3. Usability 4. Efficiency 5. Maintainability 6. Portability

Functionality
A set of attributes that bear on the existence of a set of functions and their specified properties. The functions are those that satisfy stated or implied needs. Suitability Accuracy Interoperability Compliance Security

Reliability
A set of attributes that bear on the capability of software to maintain its level of performance under stated conditions for a stated period of time. Maturity Recoverability Fault Tolerance

Usability
A set of attributes that bear on the effort needed for use, and on the individual assessment of such use, by a stated or implied set of users.
Learnability Understandability Operability

Efficiency
A set of attributes that bear on the relationship between the level of performance of the software and the amount of resources used, under stated conditions. Time Behaviour Resource Behaviour

Maintainability
A set of attributes that bear on the effort needed to make specified modifications. Stability Analyzability Changeability Testability

Portability
A set of attributes that bear on the ability of software to be transferred from one environment to another. Installability Replaceability Adaptability

Portability
A set of attributes that bear on the ability of software to be transferred from one environment to another. Installability Replaceability Adaptability

McCall's Quality Factors

Goal-Question-Metric (GQM)
A measurement program can be more successful if designed with the goals in mind. GQM approach provides a framework with 3 steps: 1. List the major goals of the development / maintenance project 2. Derive from each goal the questions that must be answered to determine if the goals are being met 3. Decide what must be measured to answer the questions adequately

Goal-Question-Metric (GQM)
BENEFITS :
generates only those measures relevant to the goal several measurements may be needed to answer a single question. single measurement may apply to more than one question. the goal provides the purpose for collecting the data.

NOT EVIDENT FROM THE GQM

The model needed to combine the measurement in a sensible way so that the question can be answered. It must be supplemented by one or more models that express the relationship among the metrics. (equation definition is not clear)

DISADVANTAGES: Additional efforts to derive the goals and metrics Error prone compared to standard models

Measuring Quality
Users view: Reliability (number of failures over time) Availability Usability etc. Often directly measurable Manufacturers view: Defect counts Rework costs

IEEE 982
Reliability is an estimation of system failure-freeness. A constructive approach to reliable software seeks to remove the root causes of this class of system failure through software development and support processes that promote fault avoidance, early fault detection, appropriately prompt removal, and system-designed fault tolerance. The analysis of the errors, faults, and failures from previous development and support processes can lead to improved future processes. While the exact functional relationships are not proven, it is through experience that the majority of failures are related to their origins. Examples include the following: 1. Incompletely defined user needs and requirements 2. Omissions in the design and coding process 3. Improper usage of the system 4. Excessive change activity

IEEE 982
Nine Classes of Measures
Product Measures
1. 2. 3. 4. 5. 6. errors, faults, failures mean-time-to-failure realibility growth and projection remaining products faults completeness and consistency complexity

Process Measures
1. management control 2. coverage 3. risk, benefit, cost evaluation

Define & give example

Usability Integrity Efficiency Correctness Reliability Maintainability Testability Flexibility Reusability Portability Interoperability

Reality Check
Q: So, how does that long list help us with SQA?
A: Most, if not all, of those factors should be covered explicitly in the software requirements document.

A: Measuring those factors tell us where we need improvement.

Operability Training Communicativeness Input/Output volume Input/Output gate Access Control Access Audit Storage efficiency Execution Efficiency Traceability Completeness Accuracy Error Tolerance Consistency Simplicity Conciseness Instrumentation Expandability Generality Self-Descriptiveness Modularity Machine Independence Software System Independence Communications Commonality Data Commonality

Usability
Integrity Efficiency Correctness Reliability Maintainability Testability Flexibility Reusability Portability Interoperability