Sie sind auf Seite 1von 56
intelligent testing
intelligent testing

An Overview for

DO-178B

Tool Qualification

intelligent testing An Overview for DO-178B Tool Qualification intelligent testing © IPL 2011

intelligent testing

© IPL 2011

Contents

Overview of and

Tool Qualification Case

Qualification Requirements and Credentials

Tool Qualification Pack

IPL Quality Involvement

Cantata++ Tool Development

Cantata++ Technical Support

Further Information

2
2

intelligent testing

© IPL 2011

intelligent business ® Trusted Consulting & Solutions Company  30 year track record  260

intelligent business ®

Trusted Consulting & Solutions Company

30 year track record

260 staff, €30m+ turnover

HQ in Bath, UK

High-stakes, business and mission critical

Real Time Systems:

Aerospace & Defence Police Telecoms Automotive & Rail Transport Medical

Police Telecoms Automotive & Rail Transport Medical intelligent testing © IPL 2011 Testing Tools:  20

intelligent testing

© IPL 2011

Testing Tools:

20 year track record Grew out of services business

intelligent testing
intelligent testing
intelligent testing intelligent testing intelligent testing © © IPL 2010 IPL 2010
intelligent testing intelligent testing intelligent testing © © IPL 2010 IPL 2010

intelligent testing

intelligent testing

intelligent testing intelligent testing intelligent testing © © IPL 2010 IPL 2010
intelligent testing intelligent testing intelligent testing © © IPL 2010 IPL 2010

intelligent testing

©

©

IPL 2010

IPL 2010

What is ?

is a software verification tool for host and target environments providing:

Dynamic Testing Unit and Integration test execution Automated results checking Test documentation / evidence of testing Test repeatability

Structural Coverage Analysis for Software Levels C to A Entry-Point, Statement, Decision, Boolean Operator, Boolean Operand Effectiveness (Masking and Unique Cause)

Static Analysis Metrics

Ancillary Tools Built on Eclipse user interface for test scripting Advanced test and coverage diagnostics

Built on Eclipse user interface for test scripting Advanced test and coverage diagnostics intelligent testing ©

intelligent testing

© IPL 2011

Unit/Integration Test – 6 Tool Challenges

CodeCode CoverageCoverage FlexibleFlexible TestTest TechniquesTechniques CallCall InterfaceInterface ControlControl
CodeCode CoverageCoverage
FlexibleFlexible TestTest TechniquesTechniques
CallCall InterfaceInterface ControlControl
TestTest HarnessHarness CreationCreation
ToolTool ReliabilityReliability
PlanningPlanning TestTest EffortEffort
Confidence that my tests on the target cover the important code Having a full tool-kit
Confidence that my tests on the target
cover the important code
Having a full tool-kit to efficiently test
the way I need to
Making it simple for me to test object
interactions with the rest of the system
Finding time to create reusable test
harnesses to test all my functionality
Not fighting the tool, but working
seamlessly with what I already have
Knowing where to focus my test effort
and how much work it will be
Developer Technical Intelligent Needs Challenges Solutions
Developer
Technical
Intelligent
Needs
Challenges
Solutions
how much work it will be Developer Technical Intelligent Needs Challenges Solutions intelligent testing © IPL

intelligent testing

© IPL 2011

how much work it will be Developer Technical Intelligent Needs Challenges Solutions intelligent testing © IPL

1 Code Complexity

Challenges:

• Analyze code before testing

• Identify maintainability

• Aid test planning

testing • Identify maintainability • Aid test planning measures… Procedural and OO Metric Sets Myers McCabe
testing • Identify maintainability • Aid test planning measures… Procedural and OO Metric Sets Myers McCabe
testing • Identify maintainability • Aid test planning measures… Procedural and OO Metric Sets Myers McCabe
testing • Identify maintainability • Aid test planning measures… Procedural and OO Metric Sets Myers McCabe

measures…

Procedural and OO Metric SetsIdentify maintainability • Aid test planning measures… Myers McCabe Halstead Hansen McCabe Object Oriented Reports

Myers McCabe Halstead Hansen McCabe Object Oriented

Reports in .csv formatSets Myers McCabe Halstead Hansen McCabe Object Oriented MOOSE MOOD QMOOD Robert Martin Bansiya’s Class Entropy

MOOSE MOOD QMOOD Robert Martin Bansiya’s Class Entropy

intelligent testing

© IPL 2011

2 Tool Reliability

Challenges:

• Reliability and Maintainability

• Market use

• Documentation, support, training etc

• Future Proofing

• Integration with environment

etc • Future Proofing • Integration with environment IPL Quality Management System LRQA Certificate 4006360

IPL Quality Management System LRQA Certificate 4006360

IPL Quality Management System LRQA Certificate 4006360 reassures with… Developed under IPL certificated QMS
IPL Quality Management System LRQA Certificate 4006360 reassures with… Developed under IPL certificated QMS

reassures with…

Developed under IPL certificated QMSManagement System LRQA Certificate 4006360 reassures with… DO-178B tool qualification package Cantata++ multi-target

DO-178B tool qualification packagereassures with… Developed under IPL certificated QMS Cantata++ multi-target deployment technology Widespread

Cantata++ multi-target deployment technologyIPL certificated QMS DO-178B tool qualification package Widespread market use Comprehensive documentation Direct

Widespread market usepackage Cantata++ multi-target deployment technology Comprehensive documentation Direct access to real engineers

Comprehensive documentationmulti-target deployment technology Widespread market use Direct access to real engineers for technical support

Direct access to real engineers for technical supporttechnology Widespread market use Comprehensive documentation On-site product training and consultancy Product road-map

On-site product training and consultancydocumentation Direct access to real engineers for technical support Product road-map intelligent testing © IPL 2011

Product road-mapaccess to real engineers for technical support On-site product training and consultancy intelligent testing © IPL

engineers for technical support On-site product training and consultancy Product road-map intelligent testing © IPL 2011

intelligent testing

© IPL 2011

3 Test Harness Creation

Challenges:

• Test harness driver for individual units

• Structured auto-repeatable test cases

• Automated checking and results production

test cases • Automated checking and results production automates through … Parse of project code Automated
test cases • Automated checking and results production automates through … Parse of project code Automated
test cases • Automated checking and results production automates through … Parse of project code Automated

automates through …

Parse of project codechecking and results production automates through … Automated structured test script case per function/ method

Automated structured test scriptproduction automates through … Parse of project code case per function/ method parameters, call order, global

case per function/ method… Parse of project code Automated structured test script parameters, call order, global data, returns test

parameters, call order, global data, returnsAutomated structured test script case per function/ method test case independence Full featured check library Positive

test case independencemethod parameters, call order, global data, returns Full featured check library Positive and negative checking

Full featured check librarycall order, global data, returns test case independence Positive and negative checking of global data Automated

Positive and negative checking of global datareturns test case independence Full featured check library Automated results production intelligent testing © IPL 2011

Automated results productionindependence Full featured check library Positive and negative checking of global data intelligent testing © IPL

check library Positive and negative checking of global data Automated results production intelligent testing © IPL

intelligent testing

© IPL 2011

4

Control of Call Interfaces

 Intercept  Simulate  Intercept
 Intercept
 Simulate
 Intercept

Challenges:

• Isolation of units from system

• Order of calls

• Different behavior on calls

• Checking/modifying parameters

behavior on calls • Checking/modifying parameters enables control by… Knowledge of calls made Automated
behavior on calls • Checking/modifying parameters enables control by… Knowledge of calls made Automated
behavior on calls • Checking/modifying parameters enables control by… Knowledge of calls made Automated

enables control by…

Knowledge of calls made• Checking/modifying parameters enables control by… Automated generation of Stubs and Isolates to simulate calls

Automated generation ofparameters enables control by… Knowledge of calls made Stubs and Isolates to simulate calls Wrappers to

Stubs and Isolates to simulate callsby… Knowledge of calls made Automated generation of Wrappers to intercept calls Programmable instances for each

Wrappers to intercept callsAutomated generation of Stubs and Isolates to simulate calls Programmable instances for each call Flexible call

Programmable instances for each calland Isolates to simulate calls Wrappers to intercept calls Flexible call sequence control Automated checks on

Flexible call sequence controlto intercept calls Programmable instances for each call Automated checks on parameters intelligent testing © IPL

Automated checks on parametersto intercept calls Programmable instances for each call Flexible call sequence control intelligent testing © IPL

intelligent testing

© IPL 2011

Stubs and Isolates A function/method inside test script with programmable instances Stub r eplaces called

Stubs and Isolates

A function/method inside test script with programmable instancesStubs and Isolates Stub r eplaces called object at link time Isolate replaces specific calls to

Stub replaces called object at link time eplaces called object at link time

Isolate replaces specific calls to called objects at link time replaces specific calls to called objects at link time

replaces specific calls to called objects at link time Source Code Check Parameters Check Call Sequences
Source Code
Source
Code

Check Parameters Check Call Sequences

Stub/Isolate for Called Object
Stub/Isolate
for Called
Object

Choose Return Parameter value(s) / Raise Exceptions

intelligent testing

© IPL 2011

Called Object
Called
Object
Wrapping A function/method inside test script with programmable instances to … select Before-After or Before-Replace

Wrapping

A function/method inside test script with programmable instances toWrapping … select Before-After or Before-Replace Pairing Intercepts specific calls to called objects at link time

select

Before-After or Before-Replace Pairing

Intercepts specific calls to called objects at link time Wrapper for External Object Check Out
Intercepts specific calls to called objects at link time
Wrapper for
External Object
Check Out Parameters
Check Call sequences
Modify Out Parameters
BEFORE
Wrapper
Source
Acts like a simulation
REPLACE
Called
Code
Wrapper
Object
AFTER
Modify In Parameters
and Return
Wrapper
Check In Parameters
and Return
© IPL 2011
intelligent testing

Call Interface Control Flexibility

Action

Stub

Isolate

Wrap

Called object simulated

Must

Must

Called object simulated Must Must

Called object included in link

-

Called object included in link - Must

Must

Check inputs to called object

Check inputs to called object
Check inputs to called object
Check inputs to called object

Check outputs for called object

-

-

Check outputs for called object - -

Check call order

Check call order
Check call order
Check call order

Check or Set any data when object called

Check or Set any data when object called
Check or Set any data when object called
Check or Set any data when object called

Check or Set any data when object returns

-

-

Check or Set any data when object returns - -

Set outputs from called object

Must

Must

Set outputs from called object Must Must

Modify inputs to called object

-

-

Modify inputs to called object - -

Modify outputs from called object

-

-

Modify outputs from called object - -

Control system calls

-

Control system calls -
Control system calls -

Control variadic function calls

Control variadic function calls -
Control variadic function calls -

-

Control class method calls

Control class method calls -

-

Control class method calls -
calls - Control variadic function calls - Control class method calls - intelligent testing © IPL

intelligent testing

© IPL 2011

5 Flexibility: Black Box Testing

Challenges:

• Large data input sets

• Checking large output sets

• Robustness tests

Test Driver Data Public  Checks
Test Driver Data
Public
 Checks
• Robustness tests Test Driver Data Public  Checks simplifies with… Table-driven test generation Multiple
• Robustness tests Test Driver Data Public  Checks simplifies with… Table-driven test generation Multiple

simplifies with…

Table-driven test generationTest Driver Data Public  Checks simplifies with… Multiple values per parameter Ranges of values Functions

Multiple values per parameter Checks simplifies with… Table-driven test generation Ranges of values Functions calculating values Combinatorial

Ranges of valuesTable-driven test generation Multiple values per parameter Functions calculating values Combinatorial effect calculator

Functions calculating valuesgeneration Multiple values per parameter Ranges of values Combinatorial effect calculator Checks on call sequences and

Combinatorial effect calculatorper parameter Ranges of values Functions calculating values Checks on call sequences and returns Robustness rule

Checks on call sequences and returnsFunctions calculating values Combinatorial effect calculator Robustness rule sets for data types intelligent testing ©

Robustness rule sets for data typescalculating values Combinatorial effect calculator Checks on call sequences and returns intelligent testing © IPL 2011

effect calculator Checks on call sequences and returns Robustness rule sets for data types intelligent testing

intelligent testing

© IPL 2011

effect calculator Checks on call sequences and returns Robustness rule sets for data types intelligent testing

5

Flexibility: White-Box Testing

Test Driver Data

Public Private Set + Check
Public
Private
Set
+
Check
Testing Test Driver Data Public Private Set + Check  Checks Challenges: • Call private methods/static
Testing Test Driver Data Public Private Set + Check  Checks Challenges: • Call private methods/static

Checks

Challenges:

• Call private methods/static functions

• Set/check private/static data

• Control of internal calls

Set/check private/static data • Control of internal calls solves the problem through… Automated accessibility
Set/check private/static data • Control of internal calls solves the problem through… Automated accessibility

solves the problem through…

Automated accessibility instrumentation• Control of internal calls solves the problem through… Call private methods and static functions directly

Call private methods and static functions directlyproblem through… Automated accessibility instrumentation Set/check data which is private, in unnamed namespaces and

Set/check data which is private, in unnamed namespaces and declared staticCall private methods and static functions directly White-box Call Interface Control Calls internal to

White-box Call Interface Controlwhich is private, in unnamed namespaces and declared static Calls internal to compilation unit Wrapping /

Calls internal to compilation unitand declared static White-box Call Interface Control Wrapping / Isolating OS library calls intelligent testing ©

Wrapping / Isolating OS library callsand declared static White-box Call Interface Control Calls internal to compilation unit intelligent testing © IPL

Interface Control Calls internal to compilation unit Wrapping / Isolating OS library calls intelligent testing ©

intelligent testing

© IPL 2011

5

Flexibility: Object Oriented Testing

Challenges:

Test case re-use aligned with code

Support for Templates

Support for Inheritance

Testing Abstract Base Classes

Test Code Code Test Code Test
Test
Code
Code
Test
Code
Test
Abstract Base Classes Test Code Code Test Code Test supports these with… Parallel hierarchy of code
Abstract Base Classes Test Code Code Test Code Test supports these with… Parallel hierarchy of code

supports these with…

Parallel hierarchy of code and test case re-useTest Code Code Test Code Test supports these with… Template instantiation Inheritance and factory methods

Template instantiationwith… Parallel hierarchy of code and test case re-use Inheritance and factory methods Abstract Base Class

Inheritance and factory methodsof code and test case re-use Template instantiation Abstract Base Class testing Object oriented code coverage

Abstract Base Class testingTemplate instantiation Inheritance and factory methods Object oriented code coverage Derived inheritance context

Object oriented code coverageInheritance and factory methods Abstract Base Class testing Derived inheritance context User contexts (State machines or

Derived inheritance contextAbstract Base Class testing Object oriented code coverage User contexts (State machines or multiple threads)

User contexts (State machines or multiple threads)Abstract Base Class testing Object oriented code coverage Derived inheritance context intelligent testing © IPL 2011

coverage Derived inheritance context User contexts (State machines or multiple threads) intelligent testing © IPL 2011

intelligent testing

© IPL 2011

5

Flexibility: Tool Configuration

Workspace Tool Preferences5 Flexibility: Tool Configuration Stored configurations Appearance of views / reports Installation details Java Coverage

Stored configurationsFlexibility: Tool Configuration Workspace Tool Preferences Appearance of views / reports Installation details Java

Appearance of views / reportsWorkspace Tool Preferences Stored configurations Installation details Java Coverage Test Script generation

Installation detailsStored configurations Appearance of views / reports Java Coverage Test Script generation Project Specific

Java CoverageAppearance of views / reports Installation details Test Script generation Project Specific Properties General

Test Script generationof views / reports Installation details Java Coverage Project Specific Properties General & specific tool

Project Specific PropertiesInstallation details Java Coverage Test Script generation General & specific tool options Advanced options Whole

General & specific tool optionsCoverage Test Script generation Project Specific Properties Advanced options Whole project or sub-folders intelligent

Advanced optionsSpecific Properties General & specific tool options Whole project or sub-folders intelligent testing © IPL 2011

Whole project or sub-foldersProject Specific Properties General & specific tool options Advanced options intelligent testing © IPL 2011

General & specific tool options Advanced options Whole project or sub-folders intelligent testing © IPL 2011

intelligent testing

General & specific tool options Advanced options Whole project or sub-folders intelligent testing © IPL 2011
General & specific tool options Advanced options Whole project or sub-folders intelligent testing © IPL 2011

© IPL 2011

6

Integrated Code Coverage

6 Integrated Code Coverage Challenges: • Set coverage targets in tests • Diagnostics over stages /

Challenges:

• Set coverage targets in tests

• Diagnostics over stages / test runs

• Sensible coverage metrics

• Coverage redundancy optimisation

coverage metrics • Coverage redundancy optimisation advanced capabilities offer… DO-178B code coverage rule
coverage metrics • Coverage redundancy optimisation advanced capabilities offer… DO-178B code coverage rule

advanced capabilities offer…

DO-178B code coverage rule sets (Levels C to A)redundancy optimisation advanced capabilities offer… Powerful drill-down views, filters and reports Coverage

Powerful drill-down views, filters and reportsoffer… DO-178B code coverage rule sets (Levels C to A) Coverage metrics Entry- Point Coverage Statement

Coverage metricsC to A) Powerful drill-down views, filters and reports Entry- Point Coverage Statement Coverage Decision Coverage

Entry- Point Coveragedrill-down views, filters and reports Coverage metrics Statement Coverage Decision Coverage (as defined by DO-178B)

Statement Coveragefilters and reports Coverage metrics Entry- Point Coverage Decision Coverage (as defined by DO-178B) Call-return

Decision Coverage (as defined by DO-178B)Coverage metrics Entry- Point Coverage Statement Coverage Call-return Coverage Condition Coverage (masking &

Call-return CoverageStatement Coverage Decision Coverage (as defined by DO-178B) Condition Coverage (masking & unique cause MC/DC)

Condition Coverage (masking & unique cause MC/DC)Coverage (as defined by DO-178B) Call-return Coverage Automatic test case coverage optimisation intelligent

Automatic test case coverage optimisationby DO-178B) Call-return Coverage Condition Coverage (masking & unique cause MC/DC) intelligent testing © IPL 2011

Coverage (masking & unique cause MC/DC) Automatic test case coverage optimisation intelligent testing © IPL 2011

intelligent testing

© IPL 2011

Test Generation Select Source Files(s) to generate Test Scripts Makefiles Test Scripts Baseline Test Generation

Test Generation

Test Generation Select Source Files(s) to generate Test Scripts Makefiles Test Scripts Baseline Test Generation
Test Generation Select Source Files(s) to generate Test Scripts Makefiles Test Scripts Baseline Test Generation

Select

Source

Files(s) to

generate

Test Scripts

Makefiles Test Scripts Baseline Test Generation Report

Makefiles

Test Scripts
Test
Scripts

Baseline Test Generation Report

Makefiles Test Scripts Baseline Test Generation Report
Test Scripts Baseline Test Generation Report Compile source with Source PARSE .csi Cantata++ Files
Compile source with Source PARSE .csi Cantata++ Files Generation Code Coverage Target Data to access
Compile source with
Source
PARSE
.csi
Cantata++
Files
Generation
Code Coverage Target
Data to access (black/white-box)
Functions to access (black/white-box)
Preferences
Call Interface Control (stub / wrap / isolate)
Baseline Testing:
• Data to Check (global, parameters, returns)
• Data to Modify (to force paths)
• Algorithm limits (time, paths, etc)

Baseline Test Algorithm

Modify (to force paths) • Algorithm limits (time, paths, etc) Baseline Test Algorithm intelligent testing ©

intelligent testing

© IPL 2011

Makefiles & Test Scripts

 Makefiles & Test Scripts Makefiles Automatically generated No sandbox copying of source code required Automated
 Makefiles & Test Scripts Makefiles Automatically generated No sandbox copying of source code required Automated

Makefiles

Automatically generated Makefiles & Test Scripts Makefiles No sandbox copying of source code required Automated CLI invocation

No sandbox copying of source code required& Test Scripts Makefiles Automatically generated Automated CLI invocation Regression Testing Script Test Test

Automated CLI invocationgenerated No sandbox copying of source code required Regression Testing Script Test Test Scripts Implemented in

Regression Testingcopying of source code required Automated CLI invocation Script Test Test Scripts Implemented in C /

Script
Script
Test
Test
Automated CLI invocation Regression Testing Script Test Test Scripts Implemented in C / C++ Multiple test

Test Scripts

Implemented in C / C++

Multiple test case typesTesting Script Test Test Scripts Implemented in C / C++ Test per function / method Table

Test per function / method Table Driven Robustness tests Object Oriented testing Baseline testing

function / method Table Driven Robustness tests Object Oriented testing Baseline testing intelligent testing © IPL

intelligent testing

© IPL 2011

Editing Tests

Editing Tests Test Script Manager Graphical view of C/C++ test script Summary Test Cases Call Interface

Test Script Manager

Graphical view of C/C++ test scriptEditing Tests Test Script Manager Summary Test Cases Call Interface Control Global Data Makefiles Synchronized Editors

SummaryTest Script Manager Graphical view of C/C++ test script Test Cases Call Interface Control Global Data

Test CasesScript Manager Graphical view of C/C++ test script Summary Call Interface Control Global Data Makefiles Synchronized

Call Interface ControlGraphical view of C/C++ test script Summary Test Cases Global Data Makefiles Synchronized Editors GUI and

Global DataC/C++ test script Summary Test Cases Call Interface Control Makefiles Synchronized Editors GUI and C/C++ test

Makefilesscript Summary Test Cases Call Interface Control Global Data Synchronized Editors GUI and C/C++ test script

Test Cases Call Interface Control Global Data Makefiles Synchronized Editors GUI and C/C++ test script Synchronize

Synchronized EditorsTest Cases Call Interface Control Global Data Makefiles GUI and C/C++ test script Synchronize on save

GUI and C/C++ test scriptInterface Control Global Data Makefiles Synchronized Editors Synchronize on save Context sensitive links External editing

Synchronize on saveMakefiles Synchronized Editors GUI and C/C++ test script Context sensitive links External editing supported

Context sensitive linksEditors GUI and C/C++ test script Synchronize on save External editing supported intelligent testing © IPL

External editing supportedSynchronized Editors GUI and C/C++ test script Synchronize on save Context sensitive links intelligent testing ©

C/C++ test script Synchronize on save Context sensitive links External editing supported intelligent testing © IPL
C/C++ test script Synchronize on save Context sensitive links External editing supported intelligent testing © IPL

intelligent testing

© IPL 2011

Test Execution Console Summary Results Explorer Test Results Summary XML Report Coverage Explorer Coverage

Test Execution

Console

Summary

Console Summary Results Explorer Test Results Summary XML Report Coverage Explorer Coverage Viewer ASCII Text Results

Results Explorer

Console Summary Results Explorer Test Results Summary XML Report Coverage Explorer Coverage Viewer ASCII Text Results

Test Results

Summary

Console Summary Results Explorer Test Results Summary XML Report Coverage Explorer Coverage Viewer ASCII Text Results

XML Report

Console Summary Results Explorer Test Results Summary XML Report Coverage Explorer Coverage Viewer ASCII Text Results

Coverage Explorer

Console Summary Results Explorer Test Results Summary XML Report Coverage Explorer Coverage Viewer ASCII Text Results

Coverage Viewer

Console Summary Results Explorer Test Results Summary XML Report Coverage Explorer Coverage Viewer ASCII Text Results

ASCII Text Results

Console Summary Results Explorer Test Results Summary XML Report Coverage Explorer Coverage Viewer ASCII Text Results

Makefiles

External Other Objects Libraries Copy Source Copy of of Copy instrumented for Testability or Coverage
External
Other
Objects
Libraries
Copy Source Copy of of
Copy instrumented for
Testability or Coverage
Source Source Code Code
Code
Test
Build
Run
Exe
C/C++
Test
Script(s)
Cantata++ Library
Cantata++
Library
Options ipg.cop
Options
ipg.cop
Code Test Build Run Exe C/C++ Test Script(s) Cantata++ Library Options ipg.cop intelligent testing © IPL

intelligent testingSource Code Code Code Test Build Run Exe C/C++ Test Script(s) Cantata++ Library Options ipg.cop ©

© IPL 2011

6.1 Supported Platforms Host Operating Systems Windows XP, Vista, 7 Linux 2.4 & 2.6.x kernel

6.1 Supported Platforms

Host Operating Systems6.1 Supported Platforms Windows XP, Vista, 7 Linux 2.4 & 2.6.x kernel 32 + 64 bit

Windows XP, Vista, 76.1 Supported Platforms Host Operating Systems Linux 2.4 & 2.6.x kernel 32 + 64 bit Host

Linux 2.4 & 2.6.x kernelPlatforms Host Operating Systems Windows XP, Vista, 7 32 + 64 bit Host Compiler Tool chains

32 + 64
32
+
64

bit

Host Compiler Tool chainsXP, Vista, 7 Linux 2.4 & 2.6.x kernel 32 + 64 bit IDEs Cantata++ supplied Eclipse

& 2.6.x kernel 32 + 64 bit Host Compiler Tool chains IDEs Cantata++ supplied Eclipse 3.6.2

IDEs

Cantata++ supplied Eclipse 3.6.22.6.x kernel 32 + 64 bit Host Compiler Tool chains IDEs Wind River Workbench 3.3 IDEs

Wind River Workbench 3.3Compiler Tool chains IDEs Cantata++ supplied Eclipse 3.6.2 IDEs compatible with Eclipse 3.6 Other proprietary IDEs

IDEs compatible with Eclipse 3.6Cantata++ supplied Eclipse 3.6.2 Wind River Workbench 3.3 Other proprietary IDEs Microsoft Visual C++ (6.0, 2003,

Other proprietary IDEsWind River Workbench 3.3 IDEs compatible with Eclipse 3.6 Microsoft Visual C++ (6.0, 2003, 2005, 2008,

Microsoft Visual C++ (6.0, 2003, 2005, 2008, 2010)3.3 IDEs compatible with Eclipse 3.6 Other proprietary IDEs GNU gcc/g++ (up to 4.6.x) Target Platforms

GNU gcc/g++ (up to 4.6.x)IDEs Microsoft Visual C++ (6.0, 2003, 2005, 2008, 2010) Target Platforms All Wind River Workbench 3.3

Target PlatformsC++ (6.0, 2003, 2005, 2008, 2010) GNU gcc/g++ (up to 4.6.x) All Wind River Workbench 3.3

All Wind River Workbench 3.3 supported platforms2005, 2008, 2010) GNU gcc/g++ (up to 4.6.x) Target Platforms Any combination of bare metal /

Any combination of bare metal / RTOS, cross-compiler and processorPlatforms All Wind River Workbench 3.3 supported platforms Cantata++ Deploy Target capability for any target platform

Cantata++ Deploy Target capability for any target platform3.3 supported platforms Any combination of bare metal / RTOS, cross-compiler and processor intelligent testing ©

cross-compiler and processor Cantata++ Deploy Target capability for any target platform intelligent testing © IPL 2011

intelligent testing

© IPL 2011

Benefits of Intelligent Testing

Benefits of Intelligent Testing intelligent testing The 6 key Technical Tool Challenges Solved by intelligent solutions
Benefits of Intelligent Testing intelligent testing The 6 key Technical Tool Challenges Solved by intelligent solutions

intelligent testing

The 6 key Technical Tool ChallengesBenefits of Intelligent Testing intelligent testing Solved by intelligent solutions + Eases tool adoption Eclipse IDE

Solved by intelligent solutionsintelligent testing The 6 key Technical Tool Challenges + Eases tool adoption Eclipse IDE & target

+

Eases tool adoptionTechnical Tool Challenges Solved by intelligent solutions + Eclipse IDE & target integration Tests in C/C++

Eclipse IDE & target integrationSolved by intelligent solutions + Eases tool adoption Tests in C/C++ Increases efficiency Automation Flexible

Tests in C/C+++ Eases tool adoption Eclipse IDE & target integration Increases efficiency Automation Flexible tool-kit Powerful

Increases efficiencyadoption Eclipse IDE & target integration Tests in C/C++ Automation Flexible tool-kit Powerful diagnostics Improves

Automation& target integration Tests in C/C++ Increases efficiency Flexible tool-kit Powerful diagnostics Improves

Flexible tool-kitintegration Tests in C/C++ Increases efficiency Automation Powerful diagnostics Improves professionalism Structured

Powerful diagnosticsin C/C++ Increases efficiency Automation Flexible tool-kit Improves professionalism Structured repeatable testing

Improves professionalismefficiency Automation Flexible tool-kit Powerful diagnostics Structured repeatable testing Knowing code will be properly

Structured repeatable testingFlexible tool-kit Powerful diagnostics Improves professionalism Knowing code will be properly tested © IPL 2011

Knowing code will be properly testedAutomation Flexible tool-kit Powerful diagnostics Improves professionalism Structured repeatable testing © IPL 2011

© IPL 2011

Advantages over the Competition

X

Advantages over the Competition X intelligent testing Integrated with Eclipse and Target Platforms Unique Call Interface

intelligent testing

Advantages over the Competition X intelligent testing Integrated with Eclipse and Target Platforms Unique Call Interface

Integrated with Eclipse and Target Platforms

testing Integrated with Eclipse and Target Platforms Unique Call Interface Control Most Automated Flexible

Unique Call Interface Control

Eclipse and Target Platforms Unique Call Interface Control Most Automated Flexible Testing Tool Kit Most Powerful

Most Automated Flexible Testing Tool Kit

Interface Control Most Automated Flexible Testing Tool Kit Most Powerful Code Coverage Diagnostics Automatic Baseline

Most Powerful Code Coverage Diagnostics

Most Automated Flexible Testing Tool Kit Most Powerful Code Coverage Diagnostics Automatic Baseline Testing © IPL

Automatic Baseline Testing

Most Automated Flexible Testing Tool Kit Most Powerful Code Coverage Diagnostics Automatic Baseline Testing © IPL

© IPL 2011

DO-1768-B Annexes

Process Objectives and Outputs by Software Level

Annex

Title

Objectives supported for levels A-D

A1

Software Planning Process

1, 2, 3, 4, 5, 6, 7

A2

Software Development Processes

1, 2, 3, 4, 5, 6, 7

A3

Verification of Outputs of Software Requirements Process

1, 2, 3, 4, 5, 6, 7

A4

Verification of Outputs of Software Design Process

1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13

A5

Verification of Outputs of Software Coding & Integration Processes

1, 2, 3, 4, 5, 6, 7

A6

Verification of Outputs of Integration Process

1, 2, 3, 4, 5

A7

Verification of Verification Process Results

1, 2, 3, 4, 5, 6, 7, 8

A8

Software Configuration Management Process

1, 2, 3, 4, 5, 6

A9

Software Quality Assurance Process

1, 2, 3

A10

Certification Liaison Process

1, 2, 3

Quality Assurance Process 1, 2, 3 A10 Certification Liaison Process 1, 2, 3 intelligent testing ©

intelligent testing

© IPL 2011

A-3 Verification of Outputs of Software Requirements Process

6 7
6
7

High level requirements are traceable to system requirements

Algorithms are accurate

Including requirements references in Cantata++ test cases

Dynamically testing the correct calculations of the algorithms.

in Cantata++ test cases Dynamically testing the correct calculations of the algorithms. intelligent testing © IPL

intelligent testing

© IPL 2011

A-4 Verification of Outputs of Software Design Process

Low level requirements are compatible with target computer

Low level requirements are verifiable

Algorithms are accurate

Software architecture is compatible with target computer

Software architecture is verifiable

Software partitioning integrity is confirmed

intelligent testingis verifiable Software partitioning integrity is confirmed Dynamically test against requirements on the target computer

Dynamically test against requirements on the target computer

Dynamically test the requirements

Dynamically test the accuracy of the algorithms.

Dynamically test architecture compatibility on the target computer

Dynamically test the architecture

Dynamically test partitioning integrity

© IPL 2011

A-5 Verification of Outputs of Software Coding & Integration Processes

1
1

Source Code complies with low level requirements

2
2

Source Code complies with software architecture

3
3

Source Code is verifiable

4
4

Source Code conforms to standards

6
6

Source Code is accurate and consistent

7
7

Output of software integration process is complete and correct

intelligent testingof software integration process is complete and correct Dynamically test against the requirements Dynamically test

Dynamically test against the requirements

Dynamically test against the architecture

Dynamically test the Source Code

Statically analyse the Source Code for standards

Dynamically test the Source Code

Dynamically test the integration

© IPL 2011

A-6 Verification of Outputs of Integration Process

1
1

Executable Object code complies with high level requirements

2
2

Executable Object Code is robust with high-level requirements

3
3

Executable Object code complies with low level requirements

4
4

Executable Object Code is robust with low-level requirements

5
5

Executable Object Code is compatible with the target computer

intelligent testingObject Code is compatible with the target computer Dynamically test against the requirements Dynamically test

Dynamically test against the requirements

Dynamically test robustness against the requirements

Dynamically test against the requirements

Dynamically test robustness against the requirements

Dynamically test compatibility on the target computer

© IPL 2011

A-7 Verification of Verification Process Results

1
1

Test procedures are correct

2
2

Test Results are correct and discrepancies explained

3
3

Test coverage of high level requirements is achieved

4
4

Test coverage of low level requirements is achieved

5
5

Test coverage of software structure (modified condition/decision)

6
6

Test coverage of software structure (decision coverage)

7
7

Test coverage of software structure (statement coverage)

8
8

Test coverage of software structure (data & control coupling)

intelligent testingcoverage of software structure (data & control coupling) Reviewable test scripts Direct from target test results

Reviewable test scripts

Direct from target test results identify passes and discrepancies

Test results include requirements references

Test results include requirements references

Masking and Unique Cause MC/DC coverage reported in test results

Decision coverage reported in test results

Statement coverage reported in test results

Dynamic checks on data read-write order & call sequence verification

© IPL 2011

DO-178B/ED-12B

Tool Qualification Case

Qualification Requirements and Credentials Tool Qualification Pack IPL Quality Involvement

Tool Development

Technical Support Further Information

IPL Quality Involvement  Tool Development  Technical Support Further Information intelligent testing © IPL 2011

intelligent testing

© IPL 2011

DO-178B

Tool Qualification Requirements

Software verification tools definition

Tools that cannot introduce errors, but may fail to detect them.” [DO-178B/ED-12B 12.2]

is a software verification tool

Qualification criteria for software verification tools

The qualification criteria for software verification tools should be achieved by demonstration that the tool complies with its Tool Operational Requirements under normal operational conditions.” [DO-178B/ED-12B 12.2.2]

Operational Requirements under normal operational conditions .” [DO-178B/ED-12B 12.2.2] intelligent testing © IPL 2011

intelligent testing

© IPL 2011

Tool Credentials

Original tool developed in 1992 to the IPL QMS

Development continued in same way from 1998

Audits

Both tools have been successfully audited by customers / DERs a number of times for use on DO-178B/ED-12B level A projects, and for various other standards

of times for use on DO-178B/ED-12B level A projects, and for various other standards intelligent testing

intelligent testing

© IPL 2011

/audits

Recent audits of /++

Software Verification Tool for use on DO-178B/ED-12B Level A projects:

April 2009: audited by TTTech, Hamilton Sundstrand, Boeing

October 2008: audited by Smiths Aerospace

January 2007: pre-audit by TTTech

June 2005: audited by ESW Wedel

April 2003 and March 2005: audited by Nord Mikro

 June 2005: audited by ESW Wedel  April 2003 and March 2005: audited by Nord

intelligent testing

© IPL 2011

Tool Qualification Pack

Version Specific CD

Tool Qualification Package Overview Qualification requirements Qualification process Links to all tool qualification data Advice of product use for DO-178B

CD Availability

Provided free of charge under Non-Disclosure Agreement

Available for customers and certification authorities

Support for use of CD in tool qualification provided by IPL

certification authorities  Support for use of CD in tool qualification provided by IPL intelligent testing

intelligent testing

© IPL 2011

Quality Involvement

Maintenance of the Quality Management System (QMS)

ensure QMS meets Company requirements

ensure QMS meets ISO9001 requirements

Ensure project meets QMS requirements

Quality involvement throughout software lifecycle

Projects subject to Quality audit

37
37

intelligent testing

© IPL 2011

QMS fits IPL’s requirements

Review

Regular and on-going

Procedures & Practices

Audits

Undertaken regularly

Conducted by Quality team

QMS documentation

Review and authorisation by Quality Review Board

Quality & Security Manager Chief Operations Officer Chief Technology Officer

Board Quality & Security Manager Chief Operations Officer Chief Technology Officer intelligent testing © IPL 2011

intelligent testing

© IPL 2011

Scope of the IPL QMS

Applies to all parts of the company including Covers all aspects of the development lifecycle:

proposals

requirements

design, implementation and software integration

project management

verification and validation

configuration management, including change management, version control and build control

process control, including enhancements

remedial, corrective and preventive actions

software maintenance and enhancement

audits and management review of the QMS

 software maintenance and enhancement  audits and management review of the QMS intelligent testing ©

intelligent testing

© IPL 2011

QMS Documentation

Project Related

Software Code of Practice (SCOP)

Quality Assurance Procedures (QAP)

Technical Support Procedures (TSP)

Others

System Management Procedures (SP)

Training Procedures (TP)

Consultancy Services Procedures (CSP)

Management Procedures (MP)

Sales and Marketing Procedures (SMP)

Career Development and Monitoring Manual (CDMM)

Sales and Marketing Procedures (SMP)  Career Development and Monitoring Manual (CDMM) intelligent testing © IPL

intelligent testing

© IPL 2011

SCOP Project Documentation

Software Quality Plan (SQP)

Defines the quality standards

Introduction To Project (ITP)

Defines the design, development and software production methods

Defines project specific requirements

How the generic QMS standards and procedures are used How any QMS standards and procedures are modified or supplemented

Others:

Project plans

Progress reports

Quality records

or supplemented Others:  Project plans  Progress reports  Quality records intelligent testing © IPL

intelligent testing

© IPL 2011

IPL QMS fits ISO’s Requirements

IPL Quality Manual

Maps IPL QMS to ISO 9001:2008

IPL Quality Function

Involvement in production/review of new QMS procedures

Undertakes on-going audit programme

External Assessment

Regular external assessments of IPL’s QMS by qualified ISO/TickIT assessor from LRQA

Last full re-assessment October 2010 by British Standards Institution

from LRQA  Last full re-assessment October 2010 by British Standards Institution intelligent testing © IPL

intelligent testing

© IPL 2011

IPL QMS Certification History

IPL QMS Certification History 1981 - 1988 - 1989 - 1992 - 1995 - 1999 -

1981

-

1988

-

1989

-

1992

-

1995

-

1999

-

2001

-

2004

-

2007

-

2009

-

2010

-

2011

-

IPL’s QMS formally documented

ISO9001 certification

- IPL’s QMS formally documented ISO9001 certification IPL QMS Current Certification LRQA 4006360 AQAP 1 &

IPL QMS Current Certification LRQA 4006360

AQAP 1 & 13 certification

ISO9001/TickIT certification

ISO9001/TickIT re-certification

ISO9001/TickIT re-certification

ISO9001/TickIT re-certification following transfer to ISO9001:2000

ISO9001/TickIT re-certification

ISO9001/TickIT re-certification

ISO9001/TickIT re-certification following transfer to ISO9001:2008

ISO9001/TickIT re-certification

Will be transitioning to TickITplus

intelligent testing

© IPL 2011

Quality Involvement - Project Lifecycle

Quality Manager

Software Quality Engineers
Software Quality Engineers
Project Lifecycle Quality Manager Software Quality Engineers liaise Products Manager Cantata++ Support Manager Activities
liaise
liaise

Products Manager

Cantata++ Support Manager

Engineers liaise Products Manager Cantata++ Support Manager Activities  Proactive project support especially on

Activities

Proactive project support especially on quality planning

Review/authorisation of documents

Project Audits

Review/witnessing/sign-off of verification and validation

Maintenance of Quality records

of verification and validation  Maintenance of Quality records intelligent testing © IPL 2011

intelligent testing

© IPL 2011

DO-178B Qualification Case Tool Development

chris.bradley@ipl.com

Development Stages of Operational Requirements System Level Testing Configuration Management

45
45

intelligent testing

© IPL 2011

A brief history of

Cantata++ 1.0 (1998) Initial release Cantata++ 2.0 (1999) - Static Analysis added Cantata++ 3.0 (2003)
Cantata++ 1.0 (1998) Initial release
Cantata++ 2.0 (1999) - Static Analysis added
Cantata++ 3.0 (2003) – Test Driver added
Cantata++ 4.0 (2004) – Studio added
Cantata++ 5.0 (2006) – Built on Eclipse
Cantata++ 6.1 (2011) – Current version
added Cantata++ 5.0 (2006) – Built on Eclipse Cantata++ 6.1 (2011) – Current version intelligent testing

intelligent testing

© IPL 2011

Development stages

Original development (to v2.0)

full development team

full development/design documentation

module tests, task tests

system acceptance tests

Maintenance (post v2.0)

Fault Reports (FRs) fixed

minor new requirements – New Feature Requests (NFRs) defined

small, incremental enhancements Project Change Request (PCRs) implemented

design documentation, module tests and task tests not maintained

system acceptance tests maintained

additional ‘extra’ tests that test all FRs, PCRs and NFRs added

tests maintained  additional ‘extra’ tests that test all FRs, PCRs and NFRs added intelligent testing

intelligent testing

© IPL 2011

Operational Requirements

IPL’s QMS demands Software Requirements Specification (SRS)

unambiguous, complete, consistent, correct

testable

configuration managed document

subject to independent and Quality review

For Requirements are defined in the ITP as

Requirements as they existed at the end of original development (corresponds to Cantata++ v2.0)

FRs and PCRs done since then (these do not affect the Cantata++ v2.0 Core Component Requirements)

NFRs done since then (these add new minor requirements)

Core Component Requirements)  NFRs done since then (these add new minor requirements) intelligent testing ©

intelligent testing

© IPL 2011

System Level Testing

The current System Level Tests comprise:

System Acceptance Tests (SATs) These are essentially the SATs as they existed at the end of original development. They test all the required functionality as of Cantata++ v2.0. The System Acceptance Test Specifications (SATSs), System Acceptance Test Procedures (SATPs) and Verification Cross-Reference Indices (VCRI) from the end of Cantata++ v2.0 apply to these

Extra tests for FRs, PCRs and NFRs Each FR, PCR and NFR has a separate Extra Test

Execution of System Level tests SATs and all Extra Tests are executed on the compilers/platform combinations documented in the Cantata++ Software Issue Form for each release.

combinations documented in the Cantata++ Software Issue Form for each release. intelligent testing © IPL 2011

intelligent testing

© IPL 2011

Configuration Management

All items configuration managed with full traceability

Documents

Code

Tests entities (configuration items) configuration managed

Changes made via formal process

FRs

PCRs (enhancements)

NFRs (new minor requirements)

Configuration managed reviews of all items modified as part of a change

Library procedures ensure

Controlled retrieval

Secure data retention

of a change Library procedures ensure  Controlled retrieval  Secure data retention intelligent testing ©

intelligent testing

© IPL 2011

Tool Qualification Case Technical Support

Delivery to Distributors / End Users Products Installation and Deployment Product Training Product Support

51
51

intelligent testing

© IPL 2011

Product Delivery

IPL Supplies Distributor or Customer

Initial product installations

License keys

Product updates

DVD or FTP

Documentation

User Guides and complete reference Manual

Hyperlinked HTML help and PDF

Further specific technical notes on Support Website

Hyperlinked HTML help and PDF  Further specific technical notes on Support Website intelligent testing ©

intelligent testing

© IPL 2011

Installation and Deployment

Installation

On development Host supported platform

Deployment

Initial / upgrade deployment of Cantata++ Library

Any necessary Cantata++ library code changes due to:

Complier restrictions Compiler bugs

Verification of correct behaviour:

Automated Cantata++ Deployment Tests

Deployment Port Report for each precise target configuration

Deployment Tests  Deployment Port Report for each precise target configuration intelligent testing © IPL 2011

intelligent testing

© IPL 2011

Product Training

Product Training Training available  Classroom product use training courses Significantly increases the productivity

Training available

Classroom product use training courses

Significantly increases the productivity of development and test engineers performing unit and integration level testing of C/C++.

Hands-on workshop covers the use of the Cantata++ product with examples and users own code.

Product orientation webinars

On-site product use consultancy and mentoring

code.  Product orientation webinars  On-site product use consultancy and mentoring intelligent testing © IPL

intelligent testing

© IPL 2011

Licensed Product Support

Customer Data

Customer details

Product delivery and installation and technical use environment

Any customer supplied code

All data securely held

Customer Enquiry

All product use enquiries replied to within 1 working day

Customer queries - Customer Enquiry Problem Report (CEPR)

Customer queries - Customer Enquiry Problem Report (CEPR) • Raised if not resolved in first contact

Raised if not resolved in first contact

May result in Fault Report (FR), Project Change Request (PCR) or workaround

CEPRs status monitored

Any faults affecting validity of results reported to all registered users

,,

intelligent testing

© IPL 2011

• Any faults affecting validity of results reported to all registered users ,, intelligent testing ©

for DO-178B

Further Information  Civil and Military Avionics Sector Briefs  Customer Case Studies  DO-178B
Further Information
 Civil and Military Avionics Sector Briefs
 Customer Case Studies
 DO-178B Tool Qualification Package CD
Sector Briefs  Customer Case Studies  DO-178B Tool Qualification Package CD intelligent testing © IPL

intelligent testing

© IPL 2011