Verification and Testing of Embedded Systems

Module 5
Non-functional Test
Bug Characteristics, Lifecycle & Reporting

Prepared By: AMIT Learning

 Outline

• Non-functional Testing

• Common Testing Terms

• Bug Definition, Characteristics, Life cycle and Reporting

Prepared By: AMIT Learning

 Intro to Performance Test

• Non-functional test aims to measuring the system behavior and performance

under loads and variant conditions.

• Requires accurate measures and massive triggers.

• Performed using specialized tools, usually cannot be held by manual test.

Prepared By: AMIT Learning

 Non-functional Testing Types

• Performance Test: evaluate the SUT performance in terms of the following:

• Speed: processing time and through put (incase of network applications).

• Accuracy: precision and losses rate

• Resources Consumption: CPU load and memory consumption.

Prepared By: AMIT Learning

 Non-functional Testing Types

• Load Test: loading the SUT gradually till the usage approaches the system’s
maximum capacity

• Stress Test: loading the SUT rapidly till the usage goes beyond the system’s
maximum capacity in an attempt to break down the system

Prepared By: AMIT Learning

 Non-functional Testing Types

• Interoperability Test: test the interaction between the different parts of the
same system, either from the same vendor or from different vendors

• Compatibility Test: check that two different systems can work properly
sharing the same platform

• Conformance Test: test the conformance to the standardized specs against

reference devices

Prepared By: AMIT Learning

 Non-functional Testing Types

• Reliability Test: test the stability of the SUT over long time of continuous
operation and under different conditions, “long time” differs from a system to
another

• Robustness Test: test the capability of the SUT to tolerate and over come
invalid inputs and conditions

Prepared By: AMIT Learning

 Non-functional Testing Types

• Configuration Test: test the behavior of the SUT under different combinations
of the possible configurations

• Usability Test: test how friendly the system is to new users, usually uses the
Test-as-you-go technique

Prepared By: AMIT Learning

 Common Testing Terms

• Monkey Test: unplanned testing, testing randomly without Test Cases

• Environmental Test: test tolerance for heat, motion, humidity, .etc

• Alpha Test: actual testing by potential user/customer at the development site

• Beta Test: actual testing by potential user/customer at the users’ site, the
product is released to a selected group of users

Prepared By: AMIT Learning

 Common Testing Terms

• Regression Test: verify that the updates in the SUT to fix a recently reported
bug or to integrate a new feature/component did not negatively impacted the
already passing TCs

• Regression Test is pretty important and exists along all of the test life
cycle

• Performed at the baselines, the builds that imply major changes, and
periodically

Prepared By: AMIT Learning

 Bug Definition

• A Bug is an error or flaw in a computer program that may prevent it from

working correctly or produce an incorrect or unintended result.

• Any anomaly from the intended system design, the organization standards,
the system functional requirements, or the non-functional requirements is
considered a bug.

Prepared By: AMIT Learning

 Characteristics of a Bug

• A bug is said to be consistent if there is a fixed procedure that reproduces

the bug every time it is executed under the same conditions

• A bug is said to be inconsistent if the reproduction procedure does not show

the bug every time it is executed under the same conditions

• It is necessary to state it if the bug is inconsistent and recommended to

associate it with the consistency ratio

Prepared By: AMIT Learning

 Components of a Bug Report

• Descriptive Summary

• Avoid vague summaries

• Avoid threatening and personal impressions!

• Keep it short and to the point

• Should help the development team to prioritize the bugs

• Bug Priority

• Prioritization indicates which bugs to be fixed first

• Usually a Technical Manager’s decision

Prepared By: AMIT Learning

 Components of a Bug Report

• Environment Setup

• Full detailed environment setup stating the effective parameters

• Contains the platform, the tools, the surrounding environment, .etc

• May refer to the setup stated in the test design document

• Build number and Feature Under Test (FUT)

Prepared By: AMIT Learning

 Components of a Bug Report

• Reproducible Procedure

• The procedure that gives the highest probability of reproducing the bug, incase it is
inconsistent

• The best bug is are producible bug

• It may refer to a typical TC procedure, only TC identification is required in this case

• Failure description

• Hit the root cause and avoid misleading aspects

• Mention the suspected component, based on the root cause

Prepared By: AMIT Learning

 Components of a Bug Report

• Bug Severity

• Critical / Blocking: An item that prevents further testing of the product or function under test
can be classified as Critical Bug. No workaround is possible for such bugs.

• Major / High: A defect that does not function as expected/designed or cause other
functionality to fail to meet requirements can be classified as Major Bug. A workaround can
be provided for such bugs.

• Average / Medium: The defects which do not conform to standards and conventions can be
classified as Medium Bugs. Easy workarounds exists to achieve functionality objectives.

• Minor / Low: Cosmetic defects which does not affect the functionality of the system can be
classified as Minor Bugs.

Prepared By: AMIT Learning

Bug Lifecycle

Prepared By: AMIT Learning

 Sample Test Report

Prepared By: AMIT Learning

 Q&A

Prepared By: AMIT Learning

THANK YOU!