Beruflich Dokumente
Kultur Dokumente
Test in the small: a test that checks a single function or class (Unit test) Test in the large: a test that checks a group of classes, such as Module test (a single module) Integration test (more than one module) System test (the entire system) Acceptance test: a formal test defined to check acceptance criteria for a software Functional test Non functional test (performance, stress test)
Software verification is often confused with software validation. The difference between verification and validation. Software verification asks the question, "Are we building the product right?"; that is, does the software conform to its specification. Software validation asks the question, "Are we building the right product?"; that is, is the software doing what the user really requires. The aim of software verification is to find the errors introduced by an activity, i.e. check if the product of the activity is as correct as it was at the beginning of the activity.
V& V goals
Verification and validation should establish confidence that the software is fit for purpose This does NOT mean completely free of defects rather, it must be good enough for its intended use and the type of use will determine the degree of confidence that is needed.
Verification Vs Validation
M.sc (IT) Page
Verification: "Are we building the product right" The software should conform to its specification Validation: "Are we building the right product" The software should do what the user really requires
Black-box testing
The black-box approach is a testing method in which test data are derived from the specified functional requirements without regard to the final program structure. It is also termed data-driven, input/output driven or requirements-based testing. Because only the functionality of the software module is of concern, black-box testing also mainly refers to functional testing -- a testing method emphasized on executing the functions and examination of their input and output data. The tester treats the software under test as a black box -- only the inputs, outputs and specification are visible, and the functionality is determined by observing the outputs to corresponding inputs. In testing, various inputs are exercised and the outputs are compared against specification to validate the correctness. All test cases are derived from the specification. No implementation details of the code are considered. It is obvious that the more we have covered in the input space, the more problems we will find and therefore we will be more confident about the quality of the software. Ideally we would be tempted to exhaustively test the input space. But as stated above, exhaustively testing the combinations of valid inputs will be impossible for most of the programs, let alone considering invalid inputs, timing, sequence, and resource variables. Combinatorial explosion is the major roadblock in functional testing. To make things worse, we can never be sure whether the specification is either correct or complete. Due to limitations of the language used in the specifications (usually natural language), ambiguity is often inevitable. Even if we use some type of formal or restricted language, we may still fail to write down all the possible cases in the specification. Sometimes, the specification itself becomes an intractable problem: it is not possible to specify precisely every situation that can be encountered using limited words. And people can seldom specify clearly what they want -- they usually can tell whether a prototype is, or is not, what they want after they have been finished. Specification problems contribute approximately 30 percent of all bugs in software. The research in black-box testing mainly focuses on how to maximize the effectiveness of testing with minimum cost, usually the number of test cases. It is not possible to exhaust the input space, but it is possible to exhaustively test a subset of the input space. Partitioning is one of the common techniques. If we have partitioned the input space and assume all the input values in a partition is equivalent, then we only need to test one representative value in each partition to sufficiently cover the whole input space. Domain testing partitions the input domain into regions, and considers the input values in each domain an equivalent class. Domains can be exhaustively tested and covered by selecting a representative value(s) in each domain. Boundary values are of special interest. Experience shows that test cases that explore boundary conditions have a higher payoff than test cases that do not. Boundary value analysis requires one or more boundary values selected as representative test cases. The difficulties with domain testing are that incorrect domain definitions in the specification cannot be efficiently discovered. Good partitioning requires knowledge of the software structure. A good testing plan will not only contain black-box testing, but also white-box approaches, and combinations of the two.
White-box testing
Contrary to black-box testing, software is viewed as a white-box, or glass-box in white-box testing, as the structure and flow of the software under test are visible to the tester. Testing plans are made according to the details of the software implementation, such as programming language, logic, and styles. Test cases are derived from the program structure. White-box testing is also called glass-box testing, logic-driven testing or design-based testing There are many techniques available in white-box testing, because the problem of intractability is eased by specific knowledge and attention on the structure of the software under test. The intention of exhausting some aspect of the software is still strong in white-box testing, and some degree of exhaustion can be achieved, such as executing each line of code at least once (statement coverage), traverse every branch statements (branch coverage), or cover all the possible combinations of true and false condition predicates (Multiple condition coverage).
M.sc (IT)
Page
Gray-Box Testing
Gray-box testing is a combination of white-box testing and black-box testing. The aim of this testing is to search for the defects if any due to improper structure or improper usage of applications. Gray-box testing is also known as translucent testing. A black-box tester is unaware of internal structure of the application to be tested, while a white-box tester knows the internal structure of the application. A gray-box tester partially knows the internal structure, which includes the access to internal structures as well as the algorithms for defining the test cases. Gray-box testers require overall and detailed description of documents with required documents of the application. Gray Box Testing collects the information for defining test cases. Gray-box testing is beneficial because it applies straight forward technique of black-box testing and influences it against the code targeted systems in white-box testing. Gray-box testing is based on requirement test case generation because it presets all the condition before program is tested by using assertion method. Requirement specification language is used to state the requirements which make easy to understand the requirements and verify its correctness too where input for requirement test case generation is the predicates and the verification discussed in requirement specification language
Performance testing
Not all software systems have specifications on performance explicitly. But every system will have implicit performance requirements. The software should not take infinite time or infinite resource to execute. "Performance bugs" sometimes are used to refer to those design problems in software that cause the system performance to degrade. Performance has always been a great concern and a driving force of computer evolution. Performance evaluation of a software system usually includes: resource usage, throughput, and stimulusresponse time and queue lengths detailing the average or maximum number of tasks waiting to be serviced by selected resources. Typical resources that need to be considered include network bandwidth requirements, CPU cycles, disk space, disk access operations, and memory usage . The goal of performance testing can be performance bottleneck identification, performance comparison and evaluation, etc. The typical method of doing performance testing is using a benchmark -- a program, workload or trace designed to be representative of the typical system usage.
Reliability testing
M.sc (IT)
Page
Security testing
Software quality, reliability and security are tightly coupled. Flaws in software can be exploited by intruders to open security holes. With the development of the Internet, software security problems are becoming even more severe. Many critical software applications and services have integrated security measures against malicious attacks. The purpose of security testing of these systems include identifying and removing software flaws that may potentially lead to security violations, and validating the effectiveness of security measures. Simulated security attacks can be performed to find vulnerabilities.
Testing automation
Software testing can be very costly. Automation is a good way to cut down time and cost. Software testing tools and techniques usually suffer from a lack of generic applicability and scalability. The reason is straight-forward. In order to automate the process, we have to have some ways to generate oracles from the specification, and generate test cases to test the target software against the oracles to decide their correctness. Today we still don't have a full-scale system that has achieved this goal. In general, significant amount of human intervention is still needed in testing. The degree of automation remains at the automated test script level. The problem is lessened in reliability testing and performance testing. In robustness testing, the simple specification and oracle: doesn't crash, doesn't hang suffices. Similar simple metrics can also be used in stress testing.
Testing Phase
"The testing phase" will refer to the phase in which comprehensive testing begins to be carried out on the development products (usually by designated testers). For discussion purposes, comprehensive testing begins from the integration stages and up to the system test. Many refer to this stage as the "QA stage". Because of the complexity of the concept "QA" and vast description, the article will adhere to the concept "testing phase". Objectives of the Testing Phase Before we examine the contribution of the testing phase to the project, it is important to understand the declared objectives of this phase. The accepted answers to this question are as follows:
M.sc (IT)
Page
"To allow interested parties in the product to receive a measure of its quality and compliance to its requirements" "To prove that the software does what it is supposed to do" (partial and problematic definition). "To find the gaps between what the software is supposed to do/not do to what it actually does/does not do" (a slightly more complete definition)
This is indeed the main purpose of the testing phase - an important and critical phase for the success of the project.
Static testing
Static testing is a system of White Box testing, where developers to verify or check your code with the help of the list to find fault with him, this type of testing is completed without executing the applications that are currently developed. Code reviews, inspections, tours are mostly completed at this stage of the check.
Dynamic testing
Dynamic Testing is completed by walking the real application with valid entries to verify the expected results. Examples of methodologies for dynamic testing are unit testing, integration testing, system testing & acceptance testing. Some differences between static & dynamic testing are
Static check is more profitable than the dynamics of the static check because tests are made at the initial stage. In terms of the declaration of coverage, the static check covers areas most dynamic tests in less time. Static Testing is completed before the implementation of code in the dynamic check is performed after the implementation of code. Static Testing is completed in the verification phase in the dynamic check is performed on the validation phase.
Testing Team
The average development life cycle of each of these components is between 3 to 4 months. Since there are multiple components being developed simultaneously, the average team size is 2, which Includes 1 developer and 1 tester. Also, the business functionalities in these components are not similar in nature, as every component is developed to meet the requirement of one financial transaction. Testing of these applications involves verification of data or information generated by each component under different business scenarios. The project has an independent testing team led by one Quality Assurance Manager, who coordinates the testing activities for the entire project. Each member of the testing team is motivated to play a role more than just a Tester. Apart from testing, the tester also works on process improvements, automations and plays a role of business expert within the project. The core competency required to test this application is the understanding of the business functionality of the components under test. In order to create more scope to acquire new business knowledge the testing team is consistently working towards improving the testing productivity. This enables them to play the role of Business Analysts as well, leading to an opportunity for growth of the individuals. One of the major challenges faced by this team is to create standardized processes and automated testing tools to test in mainframe testing environment. Since, there is no tool available in the market to test the application on the desired environment, the team has taken the initiative to build the tools and work towards consistent improvements. One of the many driving factors to these initiatives is the need for the reduction of manual work and the rework during the testing life cycle.
M.sc (IT)
Page
Code-driven testing: - The public (usually) interfaces to classes, modules or libraries are tested with a variety of input arguments to validate that the results that are returned are correct. Graphical user interface testing: - A testing framework generates user interface events such as keystrokes and mouse clicks, and observes the changes that result in the user interface, to validate that the observable behavior of the program is correct.
Test automation tools can be expensive, and it is usually employed in combination with manual testing. It can be made cost-effective in the longer term, especially when used repeatedly in regression testing. What to automate, when to automate, or even whether one really needs automation are crucial decisions which the testing (or development) team must make. Selecting the correct features of the product for automation largely determines the success of the automation. Automating unstable features or features that are undergoing changes should be avoided.
Testing Tools
M.sc (IT)
Page
Latest version
HP QuickTest Professional
HP Software Division
11.0
Open source
2.0.8
8.2.1
LabVIEW
National Instruments
2011
Maveryx
Maveryx
1.2.0
QF-Test
3.4.3
Ranorex
Ranorex GmbH
3.2.1
Rational robot
IBM Rational
2003
Selenium
Open source
2.11
SilkTest
Micro Focus
2010 R2 WS2
SOAtest
Parasoft
9.0
TestComplete
SmartBear Software
8.6
Testing Anywhere
Automation Anywhere
7.0
TestPartner
Micro Focus
6.3
TPT
PikeTec GmbH
3.4.3
TOSCA Testsuite
M.sc (IT)
2010
Page
M.sc (IT)
Page
Functional Testing
Functional testing is a type of black box testing that bases its test cases on the specifications of the software component under test. Functions are tested by feeding them input and examining the output, and internal program structure is rarely considered (Not like in white-box testing).[1] Functional testing differs from system testing in that functional testing "verif[ies] a program by checking it against ... design document(s) or specification(s)", while system testing "validate[s] a program by checking it against the published user or system requirements"(Kaner, Falk, Nguyen 1999, p. 52). Functional testing typically involves five steps [citation needed]: 1. The identification of functions that the software is expected to perform 2. The creation of input data based on the function's specifications 3. The determination of output based on the function's specifications 4. The execution of the test case 5. The comparison of actual and expected outputs
Non-Functionality Testing
Non-functional testing refers to aspects of the software that may not be related to a specific function or user action, such as scalability or other performance, behavior under certain constraints, or security. Non-functional requirements tend to be those that reflect the quality of the product, particularly in the context of the suitability perspective of its users. Non functional testing tests the characteristics of the software like how fast the response is, or what time does the software takes to perform any operation. Non-functional testing includes:
Baseline testing Compatibility testing Compliance testing Documentation testing Load testing Performance testing Recovery testing Security testing Scalability testing Stress testing Usability testing Volume testing
M.sc (IT)
Page
2.
When computer software is considered, black-box testing eludes to tests that are conducted at the software interface. Although they are designed to uncover errors, black-box tests are used to demonstrate that software functions are operational, that input is properly accepted and output incorrectly produced, and that the integrity of external information (e.g., a database) is maintained. A black-box test examines some fundamental aspect of a system with little regard for the internal logical structure of the software. White-box testing of software is predicated on close examination of procedural detail. Logical paths through the software are tested by providing test cases that exercise specific sets of conditions and/or loops. The "status of the program" may be examined at various points to determine if the expected or asserted status corresponds to the actual status. At first glance it would seem that very thorough white-box testing would lead to "100 percent correct programs." All we need do is define all logical paths, develop test cases to exercise them, and evaluate results, that is, generate test cases to exercise program logic exhaustively. Unfortunately, exhaustive testing presents certain logistical problems. For even small programs, the number of possible logical paths can be very large. For example, consider the 100 line program in the language C. After some basic data declaration, the program contains two nested loops that execute from 1 to 20 times each, depending on conditions specified at input. Inside the interior loop, four if-then-else constructs are required. There are approximately 1014 possible paths that may be executed in this program! To put this number in perspective, we assume that a magic test processor ("magic" because no such processor exists) has been developed for exhaustive testing. The processor can develop a test case, execute it, and evaluate the results in one millisecond. Working 24 hours a day, 365 days a year, the processor would work for 3130 years to test the program. This would, undeniably, cause havoc in most development schedules. Exhaustive testing is impossible for large software systems.
Requirement Document
Official statement of what is required of the system developers should include both a dentition and a speciation of requirements should: Specify external system behavior Specify implementation constraints Be easy to change(but changes must be managed) Serve as a reference tool for maintenance Record forethought about the life cycle of the system (i.e. predict changes) characterize responses to unexpected events It is not a design document It should state what the system should do rather than how it should do it.
M.sc (IT)
Page
10
Contents:
Executive summary Purpose Goals/objectives Target Market & Customer Overall position Competition Use cases/use model Customer needs & Corresponding Features Systems & Technical Requirements Quality Assurance & Testing
Contents:
Purpose Stakeholders Project scope/Requirements scope Market overview Product overview Use cases/use models Functional requirements Data requirements Interface requirements Support requirements Usability requirements Non Functional requirements
The final three documents are pretty much fully interchangeable. They are all documents that are used to provide a clear and accurate description of the technical specification and functionality of the system being developed.
M.sc (IT)
Page
11
FSD Functional Specifications Document; PSD - Product Specifications Document; SRS - Software Requirements Specification
Also known as: Functional spec, Specs, Software specification Usage: Used to identify the detailed requirements to aid in designing and developing the software Written by: Engineering Lead; Product Analyst; Program Manager
Contents:
Introduction - purpose; product overview; scope; references; Current system Summary Proposed methods and procedures Detailed characteristics Use cases Design considerations Environment Security
Requirement Types
Requirements types are logical groupings of requirements by common functions, features and attributes. There are four requirement types within three distinct requirement levels:
M.sc (IT)
Page
12
Requirements description
A corner gourmet pizza vendor, who has operated a traditional pizza delivery service using telephone orders, wants to automate the ordering process by developing an online system. The customers need to be able to: 1. Select the pizza toppings, size, and number of pizzas. 2. Log in and enter the delivery address. 3. Specify the time of delivery. 4. Revise or delete their orders. A store associate should be able to emulate a member login and perform the corresponding member functions on their behalf.
M.sc (IT)
Page
13
M.sc (IT)
Page
14
M.sc (IT)
Page
15