IEC Certification Kit Simulink Verification and Validation ISO 26262 Tool Qualification Package

R2012b

How to Contact MathWorks

www.mathworks.com comp.soft-sys.matlab www.mathworks.com/contact_TS.html suggest@mathworks.com bugs@mathworks.com doc@mathworks.com service@mathworks.com info@mathworks.com Web Newsgroup Technical Support Product enhancement suggestions Bug reports Documentation error reports Order status, license renewals, passcodes Sales, pricing, and general information

508-647-7000 (Phone) 508-647-7001 (Fax) The MathWorks, Inc. 3 Apple Hill Drive Natick, MA 01760-2098
For contact information about worldwide offices, see the MathWorks Web site.

IEC Certification Kit: Simulink Verification and Validation ISO 26262 Tool Qualification Package
COPYRIGHT 2011-2012 by The MathWorks, Inc.
The software described in this document is furnished under a license agreement. The software may be used or copied only under the terms of the license agreement. No part of this manual may be photocopied or reproduced in any form without prior written consent from The MathWorks, Inc. FEDERAL ACQUISITION: This provision applies to all acquisitions of the Program and Documentation by, for, or through the federal government of the United States. By accepting delivery of the Program or Documentation, the government hereby agrees that this software or documentation qualifies as commercial computer software or commercial computer software documentation as such terms are used or defined in FAR 12.212, DFARS Part 227.72, and DFARS 252.227-7014. Accordingly, the terms and conditions of this Agreement and only those rights specified in this Agreement, shall pertain to and govern the use, modification, reproduction, release, performance, display, and disclosure of the Program and Documentation by the federal government (or other entity acquiring for or through the federal government)and shall supersede any conflicting contractual terms or conditions. If this License fails to meet the governments needs or is inconsistent in any respect with federal procurement law, the government agrees to return the Program and Documentation, unused, to The MathWorks, Inc.

Trademarks
MATLAB and Simulink are registered trademarks of The MathWorks, Inc. See www.mathworks.com/trademarks for a list of additional trademarks. Other product or brand names may be trademarks or registered trademarks of their respective holders.

Patents
MathWorks products are protected by one or more U.S. patents. Please see www.mathworks.com/patents for more information.

Revision History September 2011 March 2012 September 2012

Online only Online only Online only

New for Version 2.0 (Applies to Release R2011b) Revised for Version 2.1 (Applies to Release R2012a) Revised for Version 3.0 (Applies to Release R2012b)

Contents
Introduction .................................................................................................................... 5 Project Identification.................................................................................................. 7 Tool Overview and Identification .............................................................................. 8 Tool Qualification Artifacts Summary ...................................................................... 9 Software Tool Criteria Evaluation Report ................................................................... 12 Tool Environment .................................................................................................... 13 Tool Configuration................................................................................................... 14 Reference Workflow ................................................................................................ 16 Tool Use Cases ........................................................................................................ 17 Generic Tool Classification ..................................................................................... 18 Software Tool Qualification Report............................................................................. 26 Requirement for Tool Qualification......................................................................... 27 Tool Qualification Documentation .......................................................................... 28 Confirmation Review of Tool Classification and Qualification .................................. 29 Requirement for Confirmation Review ................................................................... 30 Validity of Generic Tool Classification ................................................................... 31 Validity of Generic Tool Qualification .................................................................... 32 Conformance with Reference Workflow ................................................................. 33

Introduction
Project Identification Tool Overview and Identification Tool Qualification Artifacts Summary

This document constitutes the ISO 26262 Tool Qualification Package for the Simulink Verification and Validation product. This document is intended for use in the ISO 26262 tool classification and qualification process for software tools. It contains templates for the ISO 26262 tool qualification work products (see ISO 26262-8, Section 11). The applicant shall review the templates for applicability to the project under consideration, and then tailor and complete them as necessary. See also: IEC Certification Kit: Users Guide ISO 26262-8, Section 11

ISO 26262-8, Clause 11 provides provisions for software tools that the applicant uses to tailor activities or tasks required by ISO 26262. The standard outlines a two-step approach to establish the required confidence in the correct functioning of such tools: Tool classification determines the required level of confidence in the software tool. Depending on the result of the tool classification, it may or may not be necessary to carry out a formal tool qualification.

When applying this approach to a software tool, the applicant must create the following work products (see ISO 26262-8, 11.5): A software tool criteria evaluation report documenting the tool classification. A software tool qualification report documenting the tool qualification, if necessary.

Note The applicant

needs to review this template for applicability to the project under consideration and insert missing information.

Project Identification
Applicant: Project under consideration: <Insert information > < List project under consideration>

Tool Overview and Identification

Simulink Verification and Validation allows users to: Check Simulink and Stateflow models for compliance with design and coding guidelines. Identify untested portions of models using structural coverage metrics.

Tool Identification
Software Tool Simulink Verification and Validation Version (Release) 3.4 (R2012b) Tool Vendor The MathWorks, Inc. 3 Apple Hill Drive Natick, MA, 01760-2098 USA

Tool Qualification Artifacts Summary

For the Simulink Verification and Validation product, the following table lists: Prerequisites (see ISO 26262-8, 11.3.1) Supporting information (see ISO 26262-8, 11.3.2) Tool qualification work products (see ISO 26262-8, 11.5)

The tool qualification artifacts listed in the table are mapped to sections in this document and other artifacts.
Artifact Safety plan Applicable prerequisites of the lifecycle phases where software tool is used Corresponding Document(s) / Artifact(s) <Insert document title, version, and filename / link > <Insert software lifecycle phase(s) > <Insert prerequisite(s)>

Predetermined maximum <Insert ASIL> ASIL Software tool documentation Simulink Verification and Validation Users Guide Version 3.4 (R2012b) September 2012
slvnv_ug.pdf

Simulink Verification and Validation Reference Version 3.4 (R2012b) September 2012
slvnv_ref.pdf

Simulink Verification and Validation: Release Notes Version 3.4 (R2012b) September 2012
rn.pdf

Environment and constraints of the software tool

MathWorks bug report system at www.mathworks.com/support/bugreports/ <Insert information>

Artifact Software tool criteria evaluation report

Corresponding Document(s) / Artifact(s) Customized and completed Chapter 2, Software Tool Criteria Evaluation Report of IEC Certification Kit:

Simulink Verification and Validation ISO 26262 Tool Qualification Package (this document)
certkitiec_slvnv_tqp.docx

IEC Certification Kit: Simulink Verification and Validation Reference Workflow

Version 3.0 (R2012b) September 2012
certkitiec_slvnv_workflow.pdf

Certificate Z10 11 12 67052 013 December 2011

certkitiec_slvnv_certificate.pdf

Report to the certificate Z10 11 12 67052 013 Version 2.1, June 2012
certkitiec_slvnv_certreport.pdf

Software tool qualification report

Customized and completed Chapter 3, Software Tool Qualification Report of IEC Certification Kit:

Simulink Verification and Validation ISO 26262 Tool Qualification Package (this document)
certkitiec_slvnv_tqp.docx

Customized and completed IEC Certification Kit:

Simulink Verification and Validation Conformance Demonstration Template

certkitiec_slvnv_cdt.docx

Certificate Z10 11 12 67052 013 December 2011

certkitiec_slvnv_certificate.pdf

Report to the certificate Z10 11 12 67052 013 Version 2.1, June 2012
certkitiec_slvnv_certreport.pdf

10

Artifact

Corresponding Document(s) / Artifact(s)

Confirmation review of Customized and completed Chapter 4, Confirmation qualification of a software Review of Tool Classification and Qualification of IEC tool Certification Kit: Simulink Verification and Validation ISO 26262 Tool Qualification Package (this document)
certkitiec_slvnv_tqp.docx

11

12

Software Tool Criteria Evaluation Report

Tool Environment Tool Configuration Reference Workflow Tool Use Cases Generic Tool Classification

13

Tool Environment
It is assumed that Simulink Verification and Validation will be used in the following environment (see ISO 26262-8, 11.4.4.1d): <Insert operating system and other pertinent environment information>

14

Tool Configuration
It is assumed that Simulink Verification and Validation will be used with the following tool configuration (see ISO 26262-8, 11.4.4.1b). Model Coverage Analysis
Configuration Parameter Coverage Settings > Coverage Pane <Insert project-specific settings> Coverage Settings > Results Pane <Insert project-specific settings> Coverage Settings > Reporting Pane <Insert relevant configuration parameter names> Coverage Settings > Options Pane <Insert relevant configuration parameter names> Coverage Settings > Filter Pane <Insert relevant configuration parameter names> <Insert project-specific settings> <Insert project-specific settings> <Insert project-specific settings> <Insert project-specific settings> <Insert project-specific settings> Setting

15

Model Compliance Checking

Configuration Parameter Check configuration Setting By Task > Modeling Standards for ISO 26262 Display configuration management data Display model metrics and complexity report Check for unconnected objects Check for fully defined interface Check for questionable constructs Check usage of Stateflow constructs Check state machine type of Stateflow charts Check usage of Math Operations blocks Check usage of Signal Routing blocks Check usage of Logic and Bit Operations blocks Check usage of Ports and Subsystems blocks Check for model objects that do not link to requirements Check for inconsistent vector indexing methods

16

Reference Workflow
It is assumed that Simulink Verification and Validation will be used as described in the reference workflow documented in IEC Certification Kit: Simulink Verification and Validation Reference Workflow.

17

Tool Use Cases

It is assumed that Simulink Verification and Validation will be used as described by one or more of the following use cases (see ISO 26262-8, 11.4.4.1c). Additional information can be found in the reference workflow document IEC Certification Kit: Simulink Verification and Validation Reference Workflow.
[SLVNV_UC1] Static analysis of a model to verify compliance with specified modeling guidelines

The Simulink Verification and Validation tool is used to check a Simulink or Stateflow model for compliance with design and coding guidelines. The model being checked can be an executable specification, a model used for production code generation, or any other interim model created during the model elaboration phase.
[SLVNV_UC2] Automatic fixing of reported issues

Subsequent to model compliance checking, the Simulink Verification and Validation tool is used to automatically fix the reported issues. The fixes are applied to the model being checked initially.
[SLVNV_UC3] Structural coverage analysis of test cases at the model level

The Simulink Verification and Validation tool is used to determine the structural coverage that can be achieved by a set of model level test cases or to identify untested portions of a Simulink or Stateflow model. Supported model coverage metrics include: Decision coverage Condition coverage Modified condition and decision coverage (MC/DC)

Structural coverage analysis can be applied to an executable specification, a model used for production code generation, or any other interim model created during the model elaboration phase.

18

Generic Tool Classification

The tool classification for Simulink Verification and Validation was performed in a generic manner, independently from the development of a particular safety-related item or element. For the generic tool classification, the reference use cases listed in the section Tool Use Cases have been taken into account. The tool classification is based on the potential malfunctions or erroneous outputs and error prevention and detection measures listed in the following, corresponding sections. Additional information can be found in the reference workflow document IEC Certification Kit: Simulink Verification and Validation Reference Workflow.
Potential Malfunctions or Erroneous Outputs

The following potential malfunctions or erroneous outputs were taken into account as part of the tool classification process:
[SLVNV_E1] Model Compliance Checking False Negative

The modeling guideline checker incorrectly marks model as compliant.

[SLVNV_E2] Model Compliance Checking False Positive

The modeling guideline checker incorrectly marks model as non-compliant.

[SLVNV_E3] Model Compliance Checking Non interference

The modeling guideline checker contains an error, but the model to be analyzed does not invoke the erroneous portion of the tool.
[SLVNV_E4] Model Compliance Checking Incorrect hyperlinks

Hyperlinks in the analysis results contain errors.

[SLVNV_E5] Model Compliance Checking Incorrect fixing of reported issues

Automatic fixing of reported issues does not work correctly.

[SLVNV_E6] Model Coverage Analysis False Negative

The model coverage analysis incorrectly marks uncovered model elements as covered.

19

[SLVNV_E7] Model Coverage Analysis False Positive

The model coverage analysis incorrectly marks covered model elements as not covered.
[SLVNV_E8] Model Coverage Analysis Non interference

The modeling coverage analysis contains an error, but the model to be analyzed does not invoke the erroneous portion of the tool.
[SLVNV_E9] Simulink Verification and Validation Usage of incorrect input data

The modeling coverage analysis contains an error, but the model to be analyzed does not invoke the erroneous portion of the tool.
[SLVNV_E10] Simulink Verification and Validation Misinterpretation of results

User interprets correct analysis results incorrectly.

[SLVNV_E11] Simulink Verification and Validation Incorrect Tool Usage

User does not follow proper procedures when using the tool.
[SLVNV_E12] Simulink Verification and Validation Incorrect or Modified Installation

User does not follow proper procedures when installing the tool, installs the tool in an incorrect operational environment, or modifies a valid installation.

20

Error Prevention and Detection Measures

The following measures, which facilitate seamless functioning of model compliance checking and model coverage analysis capabilities of the Simulink Verification and Validation tool, are referenced in the tool classification process. Additional considerations are described in IEC Certification Kit: Simulink Verification and Validation Reference Workflow.
[M1]

Before or after static analysis of a model to verify its compliance with specified modeling guidelines: Dynamically verify (test) the model.
[M2]

After automatic fixing of reported issues, do one or more of the following: Re-check the model for its compliance with specified modeling guidelines. Dynamically verify (test) the model. Compare the XML files exported1 from the original and fixed Simulink models and manually review the comparison results.
[M3]

After carrying out model coverage analysis: Use a code coverage tool when testing the software generated from the model to determine structural coverage of test cases at the software level.

Requires Simulink Report Generator

21

Tool Classification Summary

Potential Use case(s) malfunction or erroneous output [SLVNV_E1] Model Compliance Checking False Negative TI Justification for TI Prevention and detection measures Preceding or subsequent dynamic verification (testing) of the model. TD Justification for TD TCL

[SLVNV_UC1] TI2 Incorrect analysis result could prevent modeling guidelines violations from being detected.

TD2 Static analysis tools TCL2 typically detect only a subset of the existing modeling standard violations in the model. Therefore, other process steps can not assume completeness of modeling guideline check results. Modeling standard violations do not necessarily imply incorrect models. Functional or structural testing help detect real errors in the model. The likelihood of detecting these errors by testing is considered to be medium.

[SLVNV_E2] Model Compliance Checking False Positive [SLVNV_E3] Model Compliance Checking Non Interference

[SLVNV_UC1] TI1 Nuisance only; model does not violate modeling guidelines. [SLVNV_UC1] TI1 Error in the tool; does not affect analysis results.

TCL1

TCL1

22

Potential Use case(s) malfunction or erroneous output [SLVNV_E4] Model Compliance Checking Incorrect hyperlinks

TI

Justification for TI

Prevention and detection measures -

TD

Justification for TD

TCL

[SLVNV_UC1] TI1 Nuisance only; model does not violate modeling guidelines.

TCL1

[SLVNV_E5] [SLVNV_UC1] TI2 Incorrect fixing Model could introduce Compliance error in the Checking model. Incorrect fixing of reported issues

[M2a] TD2 Re-checking of the TCL2 Subsequent model will detect re-checking modeling standard of the model violations introduced for by the automatic compliance fixing but might miss with specified other errors modeling introduced. guidelines. [M2b] Subsequent dynamic verification (testing) of the model. TD2 Functional or structural testing help detect real errors in the model. The likelihood of detecting these errors by testing is considered to be medium. TCL2

[M2c] TD1 Manual review of the TCL1 Subsequent comparison results comparison of can verify that fixing the XML files of changes resulted exported did not introduce from the unintended changes. original and fixed Simulink models and manual review of the comparison results.

23

Potential Use case(s) malfunction or erroneous output [SLVNV_E6] Model Coverage Analysis - False Negative

TI

Justification for TI

Prevention and detection measures None

TD

Justification for TD

TCL

[SLVNV_UC3] TI2 Incorrect analysis result could prevent incomplete test cases from being detected. Incomplete test cases could result in untested portions of the model or generated code.

TD3 -

TCL3

[M3] TD1 Use of a code TCL1 Subsequent coverage tool usage of a determines code coverage completeness of tests tool when at the software level. testing the software generated from the model. TCL1

[SLVNV_E7] Model Coverage Analysis - False Positive [SLVNV_E8] Model Coverage Analysis - Non interference [SLVNV_E9] Simulink Verification and Validation Usage of incorrect input data1

[SLVNV_UC3] TI1 Nuisance only; test cases are complete. [SLVNV_UC3] TI1 Error in the tool; does not affect analysis results. [SLVNV_UC1] TI2 Incorrect or [SLVNV_UC3] incomplete analysis results could prevent errors from being detected.

TCL1

[M_MISC1] TD1 Revision control and TCL1 Revision configuration control and management configuration facilitate integrity of management2 the artifacts to be to correctly verified. Using identify the checksums allows the artifacts to be unique identification verified; use the artifacts being of checksums. verified. [M_MISC2] Competency of the project team. 3 TD1 Training of users can TCL1 prevent these issues.

[SLVNV_E10] [SLVNV_UC1] TI2 Misinterpretation Simulink [SLVNV_UC2] of analysis Verification and [SLVNV_UC3] results could Validation prevent errors Misinterpretation from being of results detected.

24

Potential Use case(s) malfunction or erroneous output [SLVNV_E11] Simulink Verification and Validation Incorrect Tool Usage [SLVNV_E12] Simulink Verification and Validation Incorrect or Modified Installation

TI

Justification for TI

Prevention and detection measures [M_MISC2] Competency of the project team.

TD

Justification for TD

TCL

[SLVNV_UC1] TI2 Incorrect usage [SLVNV_UC2] could prevent [SLVNV_UC3] errors from being detected.

TD1 Training of users can TCL1 prevent these issues.

[SLVNV_UC1] TI2 Incorrect or [SLVNV_UC2] modified [SLVNV_UC3] installation could prevent errors from being detected.

[M_MISC4] TD1 Adherence to Adherence to installation guide installation and verification of guide the installed tool instructions.4 version facilitate seamless and installation. [M_MISC3] Measures to verify integrity of installed tool version.5

TCL1

2 See 3 See 4 See 5

Validation Reference Workflow. Workflow.

For example, analysis of the wrong model. Configuration Management and Revision Control of IEC Certification Kit: Simulink Verification and Competency of the Project Team of IEC Certification Kit: Simulink Verification and Validation Reference

Validation Reference Workflow.

Installation Integrity and Release Compatibility of IEC Certification Kit: Simulink Verification and

Could include re-running the validation tests shipping with the IEC Certification Kit before using Simulink Verification and Validation.

25

Based on the preceding analysis, the maximum tool impact of the Simulink Verification and Validation use cases taken into account is TI2. Applying the prevention and detection measures previously described provides a medium degree of confidence that a malfunction or an erroneous output of the model compliance checking capability of Simulink Verification and Validation can be prevented or detected. The resulting maximum required tool confidence level for model compliance checking is TCLMAX2. Applying no prevention or detection measures to verify the results of the model coverage analysis results in a maximum required tool confidence level of TCLMAX3 for the model coverage analysis capability of Simulink Verification and Validation. Subsequent use of a code coverage tool when testing the software generated from the model and the application of the generic prevention and detection measures M_MISC1, M_MISC2, M_MISC3, and M_MISC4 provides a high degree of confidence that a malfunction or an erroneous output of the modeling guidelines checking capability of Simulink Verification and Validation can be prevented or detected. In this case, the resulting maximum required tool confidence level for model coverage analysis is TCLMAX1. TV SD reviewed the generic tool classification and confirmed the preceding results in Report to the certificate Z10 11 12 67052 013.

26

Software Tool Qualification Report

Requirement for Tool Qualification Tool Qualification Documentation

27

Requirement for Tool Qualification

Given the maximum required tool confidence level TCLMAX2 for Model Compliance Checking (see Generic Tool Classification), this capability of Simulink Verification and Validation needs to be qualified up to TCL2. Permissible tool qualifications methods for TCL2 are listed in ISO 26262-8 Table 5. Given the maximum required tool confidence level TCLMAX3 for Model Coverage Analysis without verification of the analysis results (see Generic Tool Classification), this capability of Simulink Verification and Validation needs to be qualified up to TCL3. Permissible tool qualifications for TCL3 are listed in ISO 26262-8 Table 4. Given the maximum required tool confidence level TCLMAX1 for Model Coverage Analysis with subsequent use of a code coverage tool (see Generic Tool Classification), this capability of Simulink Verification and Validation does not require formal tool qualification methods (see ISO 26262-8, 11.4.6.1).

28

Tool Qualification Documentation

MathWorks carried out an application independent pre-qualification of Simulink Verification and Validation. The Model Compliance Checking capability using the ISO 26262 modeling standard checks was prequalified for all ASILs according to ISO 26262-8, up to and including TCL 2. The Model Coverage Analysis capability was prequalified for all ASILs according to ISO 26262-8, up to and including TCL 3. The pre-qualification of Simulink Verification and Validation was carried out using a combination of the following methods: Evaluation of the tool development process (ISO 26262-8, Tables 4 and 5, Method 1b). Validation of the software tool (ISO 26262-8, Tables 4 and 5, Method 1c). According to ISO 26262-8, Tables 4 and 5, these two methods are permissible for all ASILs. For TCL2, method 1b is highly recommended for ASILs A, B, and C. Method 1c is highly recommended for ASIL D. For TCL3, method 1b is highly recommended for ASILs A and B. Method 1c is highly recommended for ASILs C and D. TV SD carried out an independent tool qualification assessment. MathWorks submitted the results of the methods applied to pre-qualify Simulink Verification and Validation to TV SD. TV SD reviewed the results of the generic tool qualification for the Model Coverage Analysis and Model Compliance Checking capabilities of Simulink Verification and Validation. TV SD confirmed the results in Report to the certificate Z10 11 12 67052 013.

29

Confirmation Review of Tool Classification and Qualification

Requirement for Confirmation Review Validity of Generic Tool Classification Validity of Generic Tool Qualification Conformance with Reference Workflow

30

Requirement for Confirmation Review

The tool classification (see

31

Software Tool Criteria Evaluation Report) was carried out independently from the development of the project under consideration. Therefore, the resulting predetermined tool confidence level shall be confirmed by the applicant prior to Simulink Verification and Validation being used for the development of a particular safety-related item or element in the project under consideration (see ISO 26262-8, 11.4.2, 11.4.10). The tool qualification (see Software Tool Qualification Report) was carried out independently from the development of the application under consideration. Therefore, the resulting generic pre-qualification shall be confirmed by the applicant prior Simulink Verification and Validation being used for the development of a particular safety-related item or element for the application under consideration (see ISO 26262-8, 11.4.2, 11.4.10). The generic tool classification is based on the assumption that Simulink Verification and Validation is being used as described in the reference workflow documented in IEC Certification Kit: Simulink Verification and Validation Reference Workflow. Therefore, conformance with the reference workflow in the project under consideration shall be confirmed by the applicant.

32

Validity of Generic Tool Classification

Applicable Tool Confidence Level: <Insert TCL> <Insert results of confirmation review or reference to confirmation review documentation>

33

Validity of Generic Tool Qualification

Applicable Tool Confidence Level: <Insert TCL> <Insert results of confirmation review or reference to confirmation review documentation>

34

Conformance with Reference Workflow

Applicable Tool Confidence Level: <Insert TCL> <Insert results of confirmation review or reference to confirmation review documentation>

35