Quality Assurance and Testing - Group 3

Quality assurance and
Testing
Group 3
9th May 2011
Andre Bruintjies
Desiré Minnaar
Neil Jonkers
Adolf Lamprecht
Agenda
• Quality and the History

• Software Quality Models
• Neils Topics
• Adolf’s Topics
Andre Bruintjies
What is Quality and Quality Assurance?
Quality
• “The totality of features and characteristics of a product or service that
bear on its ability to satisfy stated or implied needs”
Computer Dictionary Online .(1995). Quality . Available: http://www.computer-dictionary-online.org/?q=quality [24 April 2011]
• “The standard of something as measured against other things of a

similar kind; the degree of excellence of something”
Oxford dictionaries. (2011). Quality. Available: http://oxforddictionaries.com/view/entry/m_en_gb0678350#m_en_gb0678350 [24 April 2011]
Quality Assurance
• A planned and systematic pattern of all actions necessary to provide
adequate confidence that the product optimally fulfils customers
expectations, i.e. that it is problem-free and will able to perform the task it
was designed for.
Computer Dictionary Online .(1995). Quality Assurance . Available: http://www.computer-dictionary-online.org/?q=quality%20assurance .[24 April 2011]
The history of quality assurance
Inspection
Total Quality
History (Evolution) of Quality Control .(2009). Available: http://www.rajeshtimane.com/56/academics/history-of-quality-control.html . [24

April 2011]
Engineering Approaches evolved to IT.
• First definition of software engineering (Bauer)

• Set of principles seeking a solution to a common problem
• Systematic planning
• Hoare’s (1975) four characteristics of software engineering
• Wassermann (1996) eight characteristics of software engineering
Yingxu Wang and Dilip Patel. (2000). Comparative Software Engineering. Available:
http://www.google.co.za/url?sa=t&source=web&cd=3&ved=0CCgQFjAC&url=http%3A%2F%2Fciteseerx.ist.psu.edu%2Fviewdoc%2Fdownload%3Fdoi%3D10.1
.1.90.9432%26rep%3Drep1%26type%3Dpdf&rct=j&q=Annals%20of%20Software%20Engineering%20Volume%2010&ei=_3bCTY3gOY3qOcO77J0I&usg=AFQj
CNGBj0m2lrlRXWC8sAacoKxfDhKR5g&sig2=ZFf3XupDB8rEIqVpRZKbng&cad=rja
[24 April 2011]
Why is quality assurance required in IT.
• Conformance to specification
• Meeting customers needs
– Change (Industry)
– Increased confidence (customers will be kept interested)
Software development life cycle
Satzinger, et. al. 2004. Systems Analysis & Design in a changing world. Thomson, United states of America.
Desire Minnaar
What is Quality ….?
• IEEE [1990] defines quality as the degree to which a system,

component or process meets specified requirements and customer
(user) needs (expectations).
• “conformance to explicitly stated functional and performance
requirements, explicitly documented development standards, and
implicit characteristics that are expected of all professionally
developed software.”
• Petrasch [1999] defines it as “the existence of characteristics of a
product which can be assigned to requirements.”
• [ISO, 1999] as “the totality of characteristics of an entity that bear on
its ability to satisfy stated and implied needs,”
• 2 Important key points:
– Conformance to specification
– Meeting customer needs:
Nasib S. G. ,(2005), “Factors affecting effective software quality management revisited”., SIGSOFT Softw. Eng. Notes 30, 2 (March 2005), pp.1-4
McCall’s Quality Model (1977)
• By Jim McCall, also known as
General Electrics Model of 1977
• Originates in the US military
• Bridges the gap between users and
developers
• 3 major perspectives for defining and
identifying quality of software product
– Product Revision
– Product Transition
– Product operations
• 11 Quality Factors (To specify)
• 23 Quality Criteria (To build)
• Metrics (To Control)
Rafa E. Al-Qutaish, PhD ,(2001), “Quality Models in Software Engineering Literature : An Analytical and Comparative Study”, Journal of American Science ,
Vol 6 ,Iss.
Milicic 3, pp.
D. 2005 166-175
, “Software Quality Attributes and Trade-Offs”, Chapter 1 - Software Quality Models and Philosophies, viewed 25 April 2100,
http://www.bth.se/tek/besq.nsf/(WebFiles)/5A52350A52726F51C12570A8004CB613/$FILE/Software_quality_attributes.pdf
McCall’s Quality Model (1977)
Product Revision Product Operations
Simplicity Tracebility
Maintainability
Conciseness
Correctness Completeness
Instrument
Testability ation
Consistency
Self-
descriptiveness Reliability Accuracy
Flexibility
Expandability Error
Tolerance
Product Transition Execution

Efficiency efficiency
Portability Generality
Storage
efficiency
Modularity
Access
Software- Control
Reusability Integrity
system
Access
independence
audit
Machine
Independance Operability
Interoperability Communication Usability

commonality Training
Data Communica
commonality tiveness
Rafa E. Al-Qutaish, PhD ,(2001), “Quality Models in Software Engineering Literature: An Analytical and Comparative Study”, Journal of American Science , Vol 6
Iss. 3, pp. 166-175
Boehm’s Quality Model (1978)
Intermediate- Distinct Primitive
Level Characteristics
Characteristics Device
Independence
• By Boehm to automatically and
quantitatively evaluate the quality Self-
Containedness
of software High-Level
Characteristics Portability
Accuracy
• Similar to McCall Quality Model:

hierarchical quality model structure Completeness
Robustness
• Consists of Reliability / Integrity
– 3 High-level characteristics General

Consistency
As-is Utility
Utility
Address the main questions of the Efficiency
Accountability
software buyer Device

Human Efficiency
– 7 Intermediate-level Engineering
Acessibility
Represent the qualities expected
Communica-
Testability tiveness
– 15 Lowest-level (primitive)
Self
Primitive characteristics provide Maintainibility Understability Descriptiveness
foundation for defining qualities Structured-

metrics Modifiability ness
Conciseness
Rafa E. Al-Qutaish, PhD ,(2001), “Quality Models in Software Engineering Literature: An Analytical and Comparative Study”, Legability
Journal of American Science , Vol 6 Iss. 3, pp. 166-175
Milicic , D. 2005 , “Software Quality Attributes and Trade-Offs”, Chapter 1 - Software Quality Models and Philosophies,
viewed 25 April 2100, Augment-
http://www.bth.se/tek/besq.nsf/(WebFiles)/5A52350A52726F51C12570A8004CB613/$FILE/Software_quality_attributes.p ability
df
ISO 9126
• Established 1991 by the International Organization for
Standardization
• Software Product Evaluation: Quality characteristics & Guidelines for
Their Use-Standard
• Based on McCall and Boehm models
• Current version has 1 International Standard, 3 Technical Reports
– ISO IS 9126 -1:Quality Model [ISO,1991]
– ISO TR 9126-2: External Metrics [ISO,2003]
– ISO TR 9126-3: Internal Metrics [ISO,2003]
– ISO TR 9126-4:Quality in Use Metrics [ISO,2004]
• Quality Model contains 2 parts
1. Internal and External quality model :
6 characteristics & 27 sub-characteristics
2. Quality in use model – 4 Quality in use characteristics
Rafa E. Al-Qutaish, PhD ,(2001), “Quality Models in Software Engineering Literature: An Analytical and Comparative Study”, Journal of American Science
, Vol 6 Iss. 3, pp. 166-175
ISO 9126: Internal External Quality Model &
Quality in Use Characteristics
Iss. 3, pp. 166-175
ISO 9126: Quality in the Lifecycle [ISO, 2001]
Iss. 3, pp. 166-175
Comparisons…
Iss. 3, pp. 166-175
ISO 9000 family of Standards
• Quality Management Process
• A series of international standards dealing with • Resource Management Process
quality systems that can be used for external • Regulatory Research Process
quality assurances purposes
• Market Research Process
• Established 1987 • Product Design Process
• Refers to 3 standards • Purchasing Process
– ISO 9000 : definitions & terminology • Production Process
– ISO 9001 : requirements for certification • Service Provision Process
– ISO 9004 : guidelines for sustained • Product Protection Process
success • Customer Needs Assessment Process
• Customer Communications Process
• ISO 9001 consists 3 main parts:
• Internal Communications Process
– Framework, describes organization aspect of a
• Document Control Process
quality system for software production
• Record Keeping Process
– Life cycle activities, defines required actions
• Planning Process
– Supporting activities, describes the minimal
• Training Process
requirements to support production, supply maintenance
• Internal Audit Process
of software
• Management Review Process
“The requirements specified are aimed primarily • Monitoring and Measuring Process
at achieving customer satisfaction • Non conformance Management Process
by preventing non-conformity at all stages • Continual Improvement Process
from design through servicing “
Milicic , D. 2005 , “Software Quality Attributes and Trade-Offs”, Chapter 1 - Software Quality Models and Philosophies, viewed 25 April 2100,
Bazzana, G.; Andersen, O.; Jokela, T. (1993), “ISO 9126 and ISO 9000: friends or foes?”, Software Engineering Standards Symposium, 1993. Proceedings., 1993 , 30 Aug-3 Sep 1993, pp 79 - 88
8 Steps to implement ISO 9001 requirements
1. Establish Commitment ( 4-12 weeks)

2. Define program structure ( 4 weeks)
3. Establish foundation elements (24 weeks)
4. Define development methodologies ( 32 weeks)
5. Define major supporting elements
6. Define remaining supporting elements
7. Preliminary Audit
– Audit (3 days)
– Correction ( up to 12 weeks)
8. Registration Audit
– Audit (3 days)
– Correction ( up to 8 weeks)
International Organization for standardisation, , ISO 9001 - Quality management systems – Requirements, viewed 2 May 2011 ,
http://www.iso.org/iso/survey2009.pdf
Helio Yang, Y. (2001) "Software quality management and ISO 9000 implementation", Industrial Management & Data Systems, Vol. 101 Iss: 7, pp.329 - 338
ISO Challenges
• The amount of money, time and paperwork required for registration

• ISO 9001 promotes specification, control, and procedures rather
than understanding and improvement
• The added cost to certify and then maintain certification may not be
justified if product end users do not require ISO 9000
• The standard is seen as especially prone to failure when a company
is interested in certification before quality. The standard is seen
as especially prone to failure when a company is interested in
certification before quality
• Another problem reported is the competition among the numerous
certifying bodies, leading to a softer approach to the defects noticed
in the operation of the Quality System of a firm.
Barnes, F , 2000, “Good Business Sense Is the Key to Confronting ISO 9000" , Review of Business, 22 March, p1-10, viewed : 03 May 2011,
http://www.allbusiness.com/specialty-businesses/713376-1.html
Seddon, J. ,(1997), "Ten arguments against ISO 9000", Managing Service Quality, Vol. 7 Iss: 4, pp.162 - 16
Other Software Quality Models & Frameworks
I SO
/I
155 EC
(SP 04
ICE
)
Milicic , D. 2005 , “Software Quality Attributes and Trade-Offs”, Chapter 1 - Software Quality Models and Philosophies, viewed 25 April 2100,
Rafa E. Al-Qutaish, PhD ,(2001), “Quality Models in Software Engineering Literature: An Analytical and Comparative Study”, Journal of American Science , Vol 6 Iss. 3, pp. 166-175
Adolf Lamprecht
Types of testing
Manual testing
According to Right Hand Technologies: Software testing is
performed to verify that the completed software package
functions according to the expectations defined by the
requirements/ specifications. The overall objective to not to find every
software bug that exists, but to uncover situations that could negatively
impact the customer, usability and/or maintainability.
According to Myers "Software Testing is the process of executing a
program or system with the intent of finding errors."
Automated testing
Definition (Ap Test, the software testing specialists):
• Testing using software tools to execute tests without manual
intervention. Can be applied in GUI, performance, API, etc. testing.
• The use of software to control the execution of tests, the comparison of
actual outcomes to predicted outcomes, the setting up of test
preconditions, and other test control and test reporting functions.
http://www.aptest.com/resources.html#app-func http:www.righthandtech.com/software-testing.php
When does manual testing make sense?
• High per-test or maintenance costs are an indicator that a test should

be done manually.
• The need for human judgment to assess the correctness of the result
or extensive, ongoing human intervention to keep the test running
– Installation, setup, operations, and maintenance like loading of
CD’s etc.
– Configuration and compatibility.
– Error handling and recovery
– Localisation
– Documentation and help
– Usability
Black, R., (2002), Investing in Software Testing: Manual or Automated? White Paper
Wild cards (auto & manual testing)
• Functional testing can either be done manually or auto

• Use cases (user scenarios)
• User interface
• Date and time handling
Higher per-test case costs and needs for human skills, judgment, and
interaction lean towards manual testing. If you need to repeat tests
many times or reduce the cycle time for test execution, lean towards
automated testing
Automated testing
According to Fewster & Graham (2000) Automated Testing is not as

easy as buying a popular test execution tool, recording the manual tests
and playing them back whenever you want to.
Fewster, M., (1999), Software Test Automation. Addison-Wesley, 1999

When does auto testing make sense?
• According to Edith et.al (2010) they have between 50000 and 100000
test cases at Ericsson per release with a time limit for testing.
• Regression and confirmation testing
• Monkey (or random).
• Load, volume, and capacity
• Performance and reliability
• Structural, especially API-based unit, component, integration
Edith, S., (2010), Towards fully automated test management for large complex systems, 2010 3rd International Conference on Software Testing,
Verification and Validation, IEEE Computer society
Reasons why test automation fails
• QA people are not programmers –smart tests
• Not enough time and resources allocated for development of
tests – inadequate planning
• Tests are fragile
• Tests are too complicated and too interdependent
• Expensive and cost of maintenance too high
• Expectations
• Lack of usefulness
• Learning curve too steep
• Poor choices of which parts to automate
• Lack of training
http://agilesoftwaretesting.com/?p=56 and http://productdevelop.blogspot.com/2010/07/some-reasons-why-test-automation.html
Types of auto testing tools
• Source testing tools (24)
• Functional testing tools (32)
• Performance testing tools (15)
• Java testing tools (27)
• Embedded testing tools (7)
• Database testing tools (13)
• Web testing tools (109)
• Bug tracking tools (66)
• Test management tools (9)
• Communication testing tools (109)
• Other (+- 25) thus in total > 300
http://www.aptest.com/resources.html#app-func
Choosing the right tool according to Fewster
(1999)
• The tool selection project
• The tool selection team
• Investigate which commercial tools are suitable for you- this order is
very important
• Start by evaluating your requirements
• Identify your constraints
• Build or buy?
• Identify what is available on the market
• Evaluate the shortlisted candidate tools
• Make the decision
Major testing tools
Tool Vendor
Loadrunner Mercury / HP
Winnrunner 10 Mercury / HP
Quick Test Professional (QTP) Mercury / HP
Silk Test / Silk Performer etc Borland formerly Segue (www.borland.com)
QA Run 4.9 Compuware (http://www.compuware.com/qarun.htm)
QA Wizard Pro 2011 Seapine Software (www.seapine.com/qawizard.html)
Visual Test Rational Software (developed by MS)
Lessons learned according to Dustin (1999)
• The various tools used throughout the development lifecycle did
not easily integrate
• Duplicate information was kept in multiple repositories
• The automated testing tool drove the testing effort.
• Everyone on the testing team was trying to automate scripts.
• Elaborate test scripts were developed, duplicating the
development effort.
• Automated test script creation was cumbersome.
• Training was too late in the process, so test engineers lacked
tool knowledge.
• Testers resisted the tool
• There were expectations of early payback
Dustin, E., (1999), Software Testing & Quality Engineering, Lessons in Test automation, Sept/Oct 1999 (Vol. 1, issue 5).
Case studies by Dr Dieter Kreuer et.al.(1999)
• Ericsson and the cellular network provider Mannesmann Mobilfunk

• Both companies seek to improve testing efficiency by means of test
automation
• Main focus was on load testing
• Test platform created by Ericsson’s and enhanced by Mannesmann
• High reuse rate of test cases make auto testing even more attractive
Kreuer, D. (1999) “Applying Test Automation to Type Acceptance Testing of Telecom Networks: A Case Study with Customer
Participation”, 14th IEEE International Conference on Automated Software Engineering (ASE'99) pp. 216-224
Forecasted development of Mannesmann CSS TAcc effort
Kreuer, D. (1999) “Applying Test Automation to Type Acceptance Testing of Telecom Networks: A Case Study with Customer
Participation”, 14th IEEE International Conference on Automated Software Engineering (ASE'99) pp. 216-224.
Why is it still so difficult to get quality right in
IT?
• Testers are ill equipped
• Ever more complex software solutions
• Testers have to cover as many aspects as possible
• Minimum criteria that testers use to judge the readiness of tested
software
• Any code specific fix can
– (a) fix only the problem that was reported
– (b) fail to fix the problem
– (c) fix the problem but break something that was previously working,
– (d) fail to fix the problem and break something else
Whittaker, J.A., What Is Software Testing? And Why Is It So Hard? IEEE Software, January/February 2000, pp. 70-79
References
Barnes, F , 2000, “Good Business Sense Is the Key to Confronting ISO 9000" , Review of Business, 22 March, p1-
10, viewed : 03 May 2011, http://www.allbusiness.com/specialty-businesses/713376-1.html
Bazzana, G.; Andersen, O.; Jokela, T. (1993), “ISO 9126 and ISO 9000: friends or foes?”, Software Engineering
Standards Symposium, 1993. Proceedings., 1993 , 30 Aug-3 Sep 1993, pp 79 – 88
Clifford S, (2005), “So many standards to follow, so little payoff’,Inc. Magazine, 1 May,p.1 viewed 03 May 2011
http://www.inc.com/magazine/20050501/management.html
Dustin, E., (1999), Software Testing & Quality Engineering, Lessons in Test automation, Sept/Oct 1999 (Vol. 1,
issue 5).
Edith, S., (2010), Towards fully automated test management for large complex systems, 2010 3 rd International
Conference on Software Testing, Verification and Validation, IEEE Computer society
Helio Yang, Y. (2001) "Software quality management and ISO 9000 implementation", Industrial Management &
Data Systems, Vol. 101 Iss: 7, pp.329 - 338
http://agilesoftwaretesting.com/?p=56
http://www.aptest.com/resources.html#app-func
http://www.ece.cmu.edu/~koopman/des_s99/sw_testing/
http:www.righthandtech.com/software-testing.php
Reference
http:www.stqemagazine.com
International Organization for standardisation, , ISO 9001 - Quality management systems –
Requirements, viewed 2 may 2011,http://www.iso.org/iso/survey2009.pdf
Milicic , D. 2005 , “Software Quality Attributes and Trade-Offs”, Chapter 1 - Software Quality Models
and Philosophies, viewed 25 April 2100,
http://www.bth.se/tek/besq.nsf/(WebFiles)/5A52350A52726F51C12570A8004CB613/$FILE/Software_quality_attri
butes.pdf
Naik, S., Tripathy ,P.,(2008), “Software Testing and Quality Assurance: Theory and Practice” , John
Wiley & Sons, INC., Hoboken, New Jersey
Nasib S. G. ,(2005), “Factors affecting effective software quality management revisited”., SIGSOFT
Softw. Eng. Notes 30, 2 (March 2005), pp.1-4
Rafa E. Al-Qutaish, PhD ,(2001), “Quality Models in Software Engineering Literature: An Analytical
and Comparative Study”, Journal of American Science , Vol 6 Iss. 3, pp. 166-175
Seddon, J. ,(1997), "Ten arguments against ISO 9000", Managing Service Quality, Vol. 7 Iss: 4,
pp.162 - 16
Kreuer, D. (1999) “Applying Test Automation to Type Acceptance Testing of Telecom
Networks: A Case
Study with Customer Participation”, 14th IEEE International Conference on Automated Software
Engineering (ASE'99) pp. 216-224.
Myers, G.J., The Art of Software Testing, Second Edition (2004), John Wiley & Sons, Inc., Hoboken,
New Jersey
Whittaker, J.A., What Is Software Testing? And Why Is It So Hard? IEEE Software, January/February 2000,
pp. 70-79
References
Barnes, F , 2000, “Good Business Sense Is the Key to Confronting ISO 9000" , Review of Business, 22 March, p1-
10, viewed : 03 May 2011, http://www.allbusiness.com/specialty-businesses/713376-1.html
Bazzana, G.; Andersen, O.; Jokela, T. (1993), “ISO 9126 and ISO 9000: friends or foes?”, Software Engineering
Standards Symposium, 1993. Proceedings., 1993 , 30 Aug-3 Sep 1993, pp 79 – 88
Clifford S, (2005), “So many standards to follow, so little payoff’,Inc. Magazine, 1 May,p.1 viewed 03 May 2011
http://www.inc.com/magazine/20050501/management.html
Helio Yang, Y. (2001) "Software quality management and ISO 9000 implementation", Industrial Management &
Data Systems, Vol. 101 Iss: 7, pp.329 - 338
International Organization for standardisation, , ISO 9001 - Quality management systems – Requirements, viewed 2
may 2011,http://www.iso.org/iso/survey2009.pdf
Milicic , D. 2005 , “Software Quality Attributes and Trade-Offs”, Chapter 1 - Software Quality Models and
Philosophies, viewed 25 April 2100,
http://www.bth.se/tek/besq.nsf/(WebFiles)/5A52350A52726F51C12570A8004CB613/$FILE/Software_quality_attrib
utes.pdf
Naik, S., Tripathy ,P.,(2008), “Software Testing and Quality Assurance: Theory and Practice” , John Wiley & Sons,
INC., Hoboken, New Jersey
Nasib S. G. ,(2005), “Factors affecting effective software quality management revisited”., SIGSOFT Softw. Eng.
Notes 30, 2 (March 2005), pp.1-4
Rafa E. Al-Qutaish, PhD ,(2001), “Quality Models in Software Engineering Literature: An Analytical and
Comparative Study”, Journal of American Science , Vol 6 Iss. 3, pp. 166-175
Seddon, J. ,(1997), "Ten arguments against ISO 9000", Managing Service Quality, Vol. 7 Iss: 4, pp.162 - 16

Quality Assurance and Testing - Group 3

Hochgeladen von

Dokumentinformationen

Copyright

Verfügbare Formate

Dieses Dokument teilen

Dokument teilen oder einbetten

Freigabeoptionen

Stufen Sie dieses Dokument als nützlich ein?

Sind diese Inhalte unangemessen?

Copyright:

Verfügbare Formate

Quality Assurance and Testing - Group 3

Hochgeladen von

Copyright:

Verfügbare Formate

Quality assurance and

• Quality and the History

• “The standard of something as measured against other things of a

History (Evolution) of Quality Control .(2009). Available: http://www.rajeshtimane.com/56/academics/history-of-quality-control.html . [24

• First definition of software engineering (Bauer)

• IEEE [1990] defines quality as the degree to which a system,

Product Transition Execution

Interoperability Communication Usability

• Similar to McCall Quality Model:

– 3 High-level characteristics General

software buyer Device

foundation for defining qualities Structured-

1. Establish Commitment ( 4-12 weeks)

• The amount of money, time and paperwork required for registration

• High per-test or maintenance costs are an indicator that a test should

• Functional testing can either be done manually or auto

According to Fewster & Graham (2000) Automated Testing is not as

Fewster, M., (1999), Software Test Automation. Addison-Wesley, 1999

• Ericsson and the cellular network provider Mannesmann Mobilfunk

Das könnte Ihnen auch gefallen