Black Box Testing

Manaj Kualitas SI / LIKMI

by
Oerip S / Rini A
 Overview

• Reviews
• Expert opinions
• Software testing
• Software maintenance
• Assurance of the quality of
the subcontractors’ work and
the customersupplied parts.
Software tests are formal SQA components
that are targeted toward review of the
actual2:25running
3/15/2018 AM of the software.http://www.if.itb.ac.id/~bayu 2
 Software Testing Strategy

• Black box testing

• White box testing

3/15/2018 2:25 AM http://www.if.itb.ac.id/~bayu 3

 Introduction Black Box Testing
• Black Box Testing focuses on functional requirements of the software
– Tester can derive sets of input conditions that will fully exercise all
functional requirements for a program.
• Black Box Testing is not an alternative to white box testing, rather it is
a complementary approach that is likely to uncover a different class of
errors than white box methods.
• Attempts to find:
– incorrect or missing functions
– interface errors
– errors in data structures or external database access
– performance errors
– initialization and termination errors.
• Tests are designed to answer the following question
– How is functionally validity tested?
– What classes of input will make good test cases?
– Is the system particularly sensitive to certain input values?
– How are the boundaries of a data class isolated
– What data rates and data volume can the system tolerate?
– What effect will specific combinations of data have on system operation?

3/15/2018 2:25 AM http://www.if.itb.ac.id/~bayu 4

 Expected Result Determination
• Expected result can not be determined by executing a sub program
• It is determined prior to test execution, at test design phase
• The cost of analyzing and predicting the expected behavior for each
test case of each subprogram is likely too high
– Use ‘oracle’
• Something which makes it possible to predict the expected result from
executing a test applying to a subprogram of a given subprogram, with
specific input data
• It may be a program, a set of values, etc
• Various categories of oracles may be applicable for difficult cases:
– Expected result is obtained by executing subprogram
• It only applies if the execution is followed up and checked (for analyzing
intermediate values and internal variables)
• Ability to prove that the result obtained are valid
• Must be used with caution
– An application may be derived from a previous already tested application
– Executing an equivalent subprogram to be tested

3/15/2018 2:25 AM http://www.if.itb.ac.id/~bayu 5

 Black Box Testing
• Equivalence Partitioning
• Boundary Value Analysis/Limit Testing
• Comparison Testing
• Sample Testing
• Robustness Testing
• Behavior Testing
• Requirement Testing
• Performance Testing
• Endurance Testing
• Cause-Effect Relationship Testing

3/15/2018 2:25 AM http://www.if.itb.ac.id/~bayu 6

 Equivalence Partitioning
• Divide the input domain into classes of data for which test cases can be
generated.
• Attempting to uncover classes of errors.
• Based on equivalence classes for input conditions.
• An equivalence class represents a set of valid or invalid states
• An input condition is either a specific numeric value, range of values, a
set of related values, or a boolean condition.
• Equivalence classes can be defined by:
 If an input condition specifies a range or a specific value, one valid and two
invalid equivalence classes defined.
 If an input condition specifies a boolean or a member of a set, one valid
and one invalid equivalence classes defined.
 Test cases for each input domain data item developed and executed.

3/15/2018 2:25 AM http://www.if.itb.ac.id/~bayu 7

 Example 1
• Requirements of the subprogram to be tested (initial&final state)
– The subprogram takes an integer input in the range
[-100,100]
– The output of the subprogram is the sign of the input value (the value 0 is
considered positive)
• Two valid equivalence class
– The input range [-100,0] which contains the subset of integer, will produce negative
sign as an output
– The input range [0,100] will produce positive sign
• Two invalid equivalence class
– The integers strictly less than -100
– The integers strictly more than -100

3/15/2018 2:25 AM http://www.if.itb.ac.id/~bayu 8

 2. Tabel pengujian
No Data Uji Data Hasil yg pengamatan kesimpulan
Input diharapk
an
1 0 <Bil Bil =50 “+” “+” OK
<=100
2 Bil > 100 Bil = 101 Muncul “+” invalid
pesan
eror
3 Bil < -100 Bil = -102 Muncul Muncul pesan OK
pesan eror
eror

4 Bil < -100 Bil = -102 Muncul “-” invalid

pesan
eror
5 -100 < Bil Bil = - 90 “–“ “–“ OK
<0

3/15/2018 2:25 AM http://www.if.itb.ac.id/~bayu 9

 Rules in Equivalence of Classes (1)
• If a condition concerning the input data specifies a range of values (e/g: the counter
can take values 1 to 999), define a valid equiv. Class [1,999] and two invalid
equivalence class (counter < 1, and counter > 999).
• If a condition concerning the input data specifies the number of values (a thing can
contain 1 to 6 name), define a valid equivalence class and two invalid equivalence
classes (no name and more than 6 names)
• If a condition concerning the input data specifies a set of values and the CSU tested
processes each value differently (e/g: BUS, TAXI, TRUCK, TRAIN), define a valid
equivalence class for each value and an invalid equivalence class (e/g
MOTORCYCLE).
• If an input condition is expressed by a sentence containing ‘must be’ (e/g the first
character must be a letter), define a valid equivalence class ( a letter) and an invalid
equivalence class ( not a letter)
• If there are reasons to believe that the CSU to be tested doesn’t process the
elements of an equivalence class in the same way, decompose the equivalence class
into smaller equivalence classes (based on a logical subprogram approach, at the
algorithm level).
– If this is not the case, this should be taken into account in defining the
equivalence classes

3/15/2018 2:25 AM http://www.if.itb.ac.id/~bayu 10

 Rules in Equivalence of Classes (2)
• If a sub program of the CSU to be tested has several inputs, the equivalence
class or test case definition process may or may not take into account of
combinatory problems.
– To test a subprogram which, has two independent input (A and B), two
valid equivalence classes (A1, A2, B1 and B2) are defined for each input
– The subprogram can be tested either by:
• Two test cases (the first case defines an input value of A1, and input
value B1, and the second value A2 and B2) and considering that these
tests are sufficient
• Or by four test cases (A1 and B1, A1 and B2, A2 and B1, A2 and B2) if
this is considered necessary.
• To test a subprogram which has two independent inputs, an invalid equivalence
class, is defined for each input
• Write a test case for each invalid equivalence class
– E/g: a requirement specifies to enter a car type, and the number of the car”
– In the test case in which the input data are two incorrect values ‘XYZ’, 0 the
program will probably not check the number of items (0), and will stop on
with the message ‘XYZ’ is not a type of car
• To test a subprogram which has several dependent input, the equivalence
classes to be defined must take into account of input combinations

3/15/2018 2:25 AM http://www.if.itb.ac.id/~bayu 11

 Example 2
• Requirements
– Given three values, represent the length of side of triangle, determine if it is
equilateral/SAMA SISI, isosceles/sama kaki and scalene /sembarang
• Three valid equivalence class
– 3 equal values positive, ( 3 sisi > 0)
– 3 positive values and 2 which are equal and the sum of two is greater than
the third value (3 sisi > 0 dan 2 sisi sama, jumlah 2)
– 3 positive values and the sum of two values is greater than the third value
(3 sisi > 0 dan jumlah 2 sisi > sisi lainnya )
• Two invalid equivalence classes
– 2 positive values and a negative/zero
– 3 positive values such that the sum of two is equal/less

3/15/2018 2:25 AM http://www.if.itb.ac.id/~bayu 12

 Boundary Value Analysis / Limit Testing
• Large number of errors tend to occur at boundaries of the input domain.
• BVA leads to selection of test cases that exercise boundary values.
• BVA complements equivalence partitioning. Rather than select any element
in an equivalence class, select those at the ''edge' of the class.
• Examples:
– For a range of values bounded by a and b, test (a-1), a, (a+1), (b-1), b, (b+1).
– If input conditions specify a number of values n, test with (n-1), n and (n+1)
input values.
– Apply 1 and 2 to output conditions (e.g., generate table of minimum and
maximum size).
– If internal program data structures have boundaries (e.g., buffer size, table
limits), use input data to exercise structures on boundaries.

3/15/2018 2:25 AM http://www.if.itb.ac.id/~bayu 13

 Comparison Testing
• In some applications the reliability of SW is critical.
– Aircraft avionics, nuclear power plant control
• Redundant hardware and software may be used to minimize error.
• For redundant SW, use separate teams to develop independent
versions of the software.
• Test each version with same test data to ensure all provide identical
output.
• Run all versions in parallel with a real-time comparison of results.
• Even if will only run one version in final system, for some critical
applications can develop independent versions and use comparison
testing or back-to-back testing.
• When outputs of versions differ, each is investigated to determine if
there is a defect.
• Method does not catch errors in the specification.

3/15/2018 2:25 AM http://www.if.itb.ac.id/~bayu 14

 Sample Testing
• Sample testing involves selecting a number of values
from an equivalence class of input data
• Integrate the values into test cases
• These values may be chosen at constant or variable
intervals

Robustness Testing
• Data are chosen outside the range defined by the
requirements
• The purpose of this test is to prove that no
catastrophic event can be produced by the
introduction of an abnormal input data item

3/15/2018 2:25 AM http://www.if.itb.ac.id/~bayu 15

 Behavior Testing
• A behavior tests is a test for which the result obtained cannot be
evaluated after a single execution of a CSU program, but can be
evaluated only after a set of linked calls to subprogram (several calls to
a given subprogram, a call to several different subprogram)
– E/g : stack

Requirement Testing
• The requirement associated with the software
(input/output/function/performance) are identified during the software
specification and design activities.
• Requirement testing involves producing a test case for each
requirement relating to CSU
• To facilitate the implementing of such tests, each requirement may be
traced to final test case trough the traceability matrix

Req. Spec. Pre-Design Det-Design CSU Test Proc. Test Rpt

3/15/2018 2:25 AM http://www.if.itb.ac.id/~bayu 16

 Performance Testing
• Performance testing evaluates the csu’s ability to operate
correctly with regard to the requirement concerning, for
example:
– data flow, memory size, execution time, etc
• To meet under certain workload or configuration conditions,
provided, however, that these requirements can be reflected at
the level of the CSU
• These requirements are defined during specification or design
activities
• Performance testing may also be used to measure and explore
the performance limits of a CSU

3/15/2018 2:25 AM http://www.if.itb.ac.id/~bayu 17

 Endurance Testing
• Endurance Testing involves repeating a given test
case a certain number of times in order to evaluate a
CSU’s ability to meet requirements.
• For example:
– the regularity in the accuraccy of certain mathematics
operations (floating point, rounding off, etc)
– management of system resources (incorrect release of
resources, etc)
– input/outputs (within the framework of the validating of an
input/output layer)
• These requirements are defined during specification
or design activities

3/15/2018 2:25 AM http://www.if.itb.ac.id/~bayu 18

 Cause-effect Relationship Testing
• This technique supplements equivalence testing by providing ways of
determining and selecting combinations of input data
• It involves representing the input state (cause) and output states
(effect) deduced from functional requirements associated with the CSU
in the form of a graph represented by logical relationship (decision
table), to avoid of having defined too many test cases
• The Steps are
– Break down the requirements into subset within which it will possible to
work
– Define cause and effect based on the requirement
– Analyze the requirements to make a logical relationship
– Highlight the graph, the impossibilities of combinations of cause/effect due
to constraint from the requirements
– Convert the graph into a decision table
• Column --> test case
• Row --> cause/effect
– Convert the columns of the decision table into test cases

3/15/2018 2:25 AM http://www.if.itb.ac.id/~bayu 19

 Example
• A CITROEN car will be identified by a letter A, B or C, and have the
letter X as the second character. Messages M1 and M2 must be sent in
the event of an error in the first or second character respectively. If the
identifier is correct, it is inserted in the database.
• The the input state (the causes) and the output states (the effect) can
be determined:
Input states: Output states:
1. 1st character: A A. database insertion
2. 1st character: B B. message M1
3. 1st character: C C. message M2
4. 2nd character: X
1 2 3 4 5 6
not
1 B
1 0 1 0 0 0
or 2 0 0 1 0
2 11 0
3 0 0 0 1
and A 0
3
4 1 1 1 0 1
A 1 1 1
not
4 C
B 1 1
C 1

3/15/2018 2:25 AM http://www.if.itb.ac.id/~bayu 20