Metrics for Testing

What is a Metric? Metric is a measure to quantify software, software development resources, and/or the software development process. A Metric can quantify any of the following factors: Schedule, or! "ffort, #roduct Si$e, #ro%ect Status, and &uality #erformance

Measuring enables. Metrics ena'les estimation of future wor!. (hat is, considering the case of testing ) *eciding the product is fit for shipment or delivery depends on the rate the defects are found and fi+ed. *efect collected and fi+ed is one !ind of metric. ,www.processimpact.comAs defined in the M.S/A /eport, .t is 'eneficial to classify metrics according to their usage. .""" 012.3 456 identifies two classes: iii#rocess 7 Activities performed in the production of the Software #roduct 7 An output of the #rocess, for e+ample the software or its documentation.

*efects are analy$ed to identify which are the ma%or causes of defect and which is the phase that introduces most defects. (his can 'e achieved 'y performing #areto analysis of defect causes and defect introduction phases. (he main requirements for any of these analysis is Software *efect Metrics. 8ew of the *efect Metrics are: Defect Density: ,9o. :f *efects /eported 'y S&A ; 9o. *efects /eported <y #eer /eview-/Actual Si$e. (he Si$e can 'e in =>:?, S>:?, or 8unction #oints. (he method used in the :rgani$ation to measure the si$e of the Software #roduct. (he S&A is considered to 'e the part of the Software testing team. Test effectiveness: t / ,t;@at- where tAtotal no. of defects reported during

testing and @at A total no. of defects reported during @ser acceptance testing

@ser Acceptance (esting is generally carried out using the Acceptance (est ?riteria according to the Acceptance (est #lan. Defect Removal Efficiency: ,(otal 9o :f *efects /emoved /(otal 9o. :f *efects .n%ected-B3CC at various stages of S*>? Description (his metric will indicate the effectiveness of the defect identification and removal in stages for a given pro%ect Formula /equirements: */" A 4,/equirement defects corrected during /equirements phase- / ,/equirement defects in%ected during /equirements phase-6 B 3CC *esign: */" A 4,*esign defects corrected during *esign phase- / ,*efects identified during /equirements phase ; *efects in%ected during *esign phase-6 B 3CC ?ode: */" A 4,?ode defects corrected during ?oding phase- / ,*efects identified during /equirements phase ; *efects identified during *esign phase ; *efects in%ected during coding phase-6 B 3CC :verall: */" A 4,(otal defects corrected at all phases 'efore delivery- / ,(otal defects detected at all phases 'efore and after delivery-6 B 3CC Metric Representation #ercentage Calculated at Stage completion or #ro%ect ?ompletion Calculated from <ug /eports and #eer /eview /eports Defect Distribution: #ercentage of (otal defects *istri'uted across

/equirements Analysis, *esign /eviews, ?ode /eviews, @nit (ests, .ntegration (ests, System (ests, @ser Acceptance (ests, /eview 'y #ro%ect >eads and #ro%ect Managers. oft!are "rocess Metrics are measures which provide information a'out the performance of the development process itself. "urpose: 3. #rovide an .ndicator to the @ltimate &uality of Software 'eing #roduced

1. Assists to the :rgani$ation to improve its development process 'y Dighlighting areas of .nefficiency or error)prone areas of the process. oft!are "roduct Metrics are measures of some attri'ute of the Software #roduct. ,"+ample, Source ?ode-. "urpose: 3. @sed to assess the quality of the output What are the most general metrics? Re#uirements Management Metrics Collected 3. 1. E. /equirements 'y state 7 Accepted, /e%ected, #ostponed 9o. of 'aselined requirements 9um'er of requirements modified after 'ase lining 3. 1. /equirements Sta'ility .nde+ ,/S./equirements to *esign (racea'ility Derived Metrics 3. Schedule Fariance 3. "ffort Fariance 3. ?ost Fariance 3. Si$e Fariance

Derived Metrics

"ro$ect Management Metrics Collected 3. #lanned 9o. of days 1. Actual 9o. of days 3. "stimated effort 1. 3. 1. 3. Actual "ffort "stimated ?ost Actual ?ost "stimated Si$e

1. Actual Si$e Testing % Revie! Metrics Collected 3. 1. E. 5. 9o. of defects found 'y /eviews 9o. of defects found 'y (esting 9o. of defects found 'y ?lient (otal 9o. of defects found 'y /eviews

Derived Metrics 3. :verall /eview "ffectiveness ,:/"1. :verall (est "ffectiveness "eer Revie!s Metrics Collected

3. 1. E. 5. G. H. I. 2. 0.

=>:? / 8# per person hour ,>anguage- for #reparation =>:? / 8# per person hour ,>anguage- for /eview Meeting 9o. of pages / hour reviewed during preparation Average num'er of defects found 'y /eviewer during #reparation 9o. of pages / hour reviewed during /eview Meeting Average num'er of defects found 'y /eviewer during /eview Meeting /eview (eam Si$e Fs *efects /eview speed Fs *efects Ma%or defects found during /eview Meeting

3C. *efects Fs /eview "ffort Derived Metrics 3. /eview "ffectiveness ,Ma%or1. (otal num'er of defects found 'y reviews for a pro%ect &ther Metrics Metrics Collected 3. 9o. of /equirements *esigned 1. 9o. of /equirements not *esigned E. 9o. of *esign elements matching /equirements 5. 9o. of *esign elements not matching /equirements G. 9o. of /equirements (ested H. 9o. of /equirements not (ested I. 9o. of (est ?ases with matching /equirements 2. 9o. of (est ?ases without matching /equirements 0. 9o. of *efects 'y Severity 3C. 9o. of *efects 'y stage of ) :rigin, *etection, /emoval Derived Metrics 3. *efect *ensity 1. 9o. of /equirements *esigned Fs not *esigned E. 9o. of /equirements (ested Fs not (ested 5. *efect /emoval "fficiency ,*/"ome Metrics E'plained Schedule Variance (SV) Description (his metric gives the variation of actual schedule vs. the planned schedule. (his is calculated for each pro%ect 7 stage wise Formula SF A 4,Actual no. of days 7 #lanned no. of days- / #lanned no. of days6 B 3CC

Metric Representation #ercentage Calculated at Stage completion Calculated from Software #ro%ect #lan for planned num'er of days for completing each stage and for actual num'er of days ta!en to complete each stage Defect Removal Efficiency (DRE) Description (his metric will indicate the effectiveness of the defect identification and removal in stages for a given pro%ect Formula /equirements: */" A 4,/equirement defects corrected during /equirements phase- / ,/equirement defects in%ected during /equirements phase-6 B 3CC *esign: */" A 4,*esign defects corrected during *esign phase- / ,*efects identified during /equirements phase ; *efects in%ected during *esign phase-6 B 3CC ?ode: */" A 4,?ode defects corrected during ?oding phase- / ,*efects identified during /equirements phase ; *efects identified during *esign phase ; *efects in%ected during coding phase-6 B 3CC :verall: */" A 4,(otal defects corrected at all phases 'efore delivery- / ,(otal defects detected at all phases 'efore and after delivery-6 B 3CC Metric Representation #ercentage Calculated at Stage completion or #ro%ect ?ompletion Calculated from <ug /eports and #eer /eview /eports Overall Revie Description (his metric will indicate the effectiveness of the /eview process in identifying the defects for a given pro%ect Formula :verall /eview "ffectiveness: :/" A 4,9um'er of defects found 'y reviews- / ,(otal num'er of defects found 'y reviews ; 9um'er of defects Effectiveness

found during (esting ; 9um'er of defects found during post)delivery-6 B 3CC Metric Representation #ercentage

Calculated at Monthly Stage completion or #ro%ect ?ompletion

Calculated from #eer reviews, 8ormal /eviews (est /eports ?ustomer .dentified *efects

Overall !est Effectiveness (O!E) Description (his metric will indicate the effectiveness of the (esting process in identifying the defects for a given pro%ect during the testing stage Formula :verall (est "ffectiveness: :(" A 4,9um'er of defects found during testing- / ,(otal num'er of defects found during (esting ; 9um'er of defects found during post delivery-6 B 3CC Metric Representation #ercentage

Calculated at Monthly <uild completion or #ro%ect ?ompletion

Calculated from (est /eports ?ustomer .dentified *efects

Effort Variance (EV) Description (his metric gives the variation of actual effort vs. the estimated effort. (his is calculated for each pro%ect Stage wise Formula "F A 4,Actual person hours 7 "stimated person hours- / "stimated person hours6 B 3CC Metric Representation

#ercentage

Calculated at Stage completion as identified in S##

Calculated from "stimation sheets for estimated values in person hours, for each activity within a given stage and Actual "ost Variance ("V) Description (his metric gives the variation of actual cost Fs the estimated cost. (his is calculated for each pro%ect Stage wise Formula ?F A 4,Actual ?ost 7 "stimated ?ost- / "stimated ?ost6 B 3CC or!ed Dours values in person hours.

Metric Representation #ercentage

Calculated at Stage completion

Calculated from "stimation sheets for estimated values in dollars or rupees, for each activity within a given stage Actual cost incurred

Si#e Variance Description (his metric gives the variation of actual si$e Fs. the estimated si$e. (his is calculated for each pro%ect stage wise Formula Si$e Fariance A 4,Actual Si$e 7 "stimated Si$e- / "stimated Si$e6 B 3CC

Metric Representation #ercentage

Calculated at Stage completion #ro%ect ?ompletion

Calculated from "stimation sheets for estimated values in 8unction #oints or =>:? Actual si$e

$roductivity on Revie Description

$re%aration & !echnical

(his metric will indicate the effort spent on preparation for /eview. @se this to calculate for languages used in the #ro%ect Formula For every language (such as C) C**) +ava) , -) etcJ. used) calculate ,=>:? or 8# - / hour ,B >anguage-

B>anguage 7 ?, ?;;, Kava, LM>, etcJ Metric Representation =>:? or 8# per hour

Calculated at Monthly <uild completion

Calculated from #eer /eview /eport

/umber of defects found per Revie! Meeting Description (his metric will indicate the num'er of defects found during the /eview Meeting across various stages of the #ro%ect Formula 9um'er of defects per /eview Meeting

Metric Representation *efects / /eview Meeting

Calculated at Monthly ?ompletion of /eview

Calculated from #eer /eview /eport #eer /eview *efect >ist i0e 1s Defects Trend.

Revie! Team Efficiency (Revie! Team Description

(his metric will indicate the /eview (eam si$e and the defects trend. (his will help to determine the efficiency of the /eview (eam Formula

/eview (eam Si$e to the *efects trend

Metric Representation /atio

Calculated at Monthly ?ompletion of /eview

Calculated from #eer /eview /eport #eer /eview *efect >ist

Revie! Effectiveness Description (his metric will indicate the effectiveness of the /eview process Formula /eview "ffectiveness A 4,9um'er of defects found 'y /eviews- / ,,(otal num'er of defects found 'y reviews- ; (esting-6 B 3CC Metric Representation #ercentage

Calculated at ?ompletion of /eview or ?ompletion of (esting stage

Calculated from #eer /eview /eport #eer /eview *efect >ist <ugs /eported 'y (esting

Total number of defects found by Revie!s Description (his metric will indicate the total num'er of defects identified 'y the /eview process. (he defects are further categori$ed as Digh, Medium or >ow Formula (otal num'er of defects identified in the #ro%ect Metric Representation *efects per Stage

Calculated at ?ompletion of /eviews

Calculated from #eer /eview /eport

#eer /eview *efect >ist

Defects vs2 Revie! effort 3 Revie! 4ield Description (his metric will indicate the effort e+pended in each stage for reviews to the defects found Formula *efects / /eview effort

Metric Representation *efects / /eview effort

Calculated at ?ompletion of /eviews

Calculated from #eer /eview /eport #eer /eview *efect >ist tability 5nde' (R 5.

Re#uirements Description

(his metric gives the sta'ility factor of the requirements over a period of time, after the requirements have 'een mutually agreed and 'aselined 'etween .vesia Solutions and the ?lient Formula /S. A 3CC B 4 ,9um'er of 'aselined requirements- 7 ,9um'er of changes in requirements after the requirements are 'aselined- 6 / ,9um'er of 'aselined requirementsMetric Representation #ercentage

Calculated at Stage completion and #ro%ect completion

Calculated from ?hange /equest Software /equirements Specification tate

Change Re#uests by Description

(his metric provides the analysis on state of the requirements

Formula 9um'er of accepted requirements 9um'er of re%ected requirements 9um'er of postponed requirements

Metric Representation 9um'er

Calculated at Stage completion

Calculated from ?hange /equest Software /equirements Specification

Re#uirements to Design Traceability Description (his metric provides the analysis on the num'er of requirements designed to the num'er of requirements that were not designed Formula (otal 9um'er of /equirements 9um'er of /equirements *esigned 9um'er of /equirements not *esigned

Metric Representation 9um'er

Calculated at Stage completion

Calculated from S/S *etail *esign

Design to Re#uirements Traceability Description (his metric provides the analysis on the num'er of design elements matching requirements to the num'er of design elements not matching requirements Formula 9um'er of *esign elements 9um'er of *esign elements matching /equirements 9um'er of *esign elements not matching /equirements

Metric Representation

9um'er

Calculated at Stage completion

Calculated from S/S *etail *esign

Re#uirements to Test case Traceability Description (his metric provides the analysis on the num'er of requirements tested Fs the num'er of requirements not tested Formula 9um'er of /equirements 9um'er of /equirements (ested 9um'er of /equirements not (ested

Metric Representation 9um'er

Calculated at Stage completion

Calculated from S/S *etail *esign (est ?ase Specification

Test cases to Re#uirements traceability Description (his metric provides the analysis on the num'er of test cases matching requirements Fs the num'er of test cases not matching requirements Formula 9um'er of /equirements 9um'er of (est cases with matching /equirements 9um'er of (est cases not matching /equirements

Metric Representation 9um'er

Calculated at Stage completion

Calculated from

S/S (est ?ase Specification

/umber of defects in coding found during testing by severity Description (his metric provides the analysis on the num'er of defects 'y the severity Formula 9um'er of *efects 9um'er of defects of low priority 9um'er of defects of medium priority 9um'er of defects of high priority

Metric Representation 9um'er

Calculated at Stage completion

Calculated from <ug /eport tage of origin) detection) removal

Defects 3

Description (his metric provides the analysis on the num'er of defects 'y the stage of origin, detection and removal. Formula 9um'er of *efects Stage of origin Stage of detection Stage of removal

Metric Representation 9um'er

Calculated at Stage completion

Calculated from <ug /eport

Defect Density Description (his metric provides the analysis on the num'er of defects to the si$e of the wor! product

Formula *efect *ensity A 4(otal no. of *efects / Si$e ,8# / =>:?-6 B 3CC Metric Representation #ercentage

Calculated at Stage completion

Calculated from *efects >ist <ug /eport

6o! do you determine metrics for your application? :'%ectives of Metrics are not only to measure 'ut also understand the progress to the :rgani$ational Moal. (he #arameters for determining the Metrics for an application: *uration ?omple+ity (echnology ?onstraints #revious "+perience in Same (echnology <usiness *omain ?larity of the scope of the pro%ect

:ne interesting and useful approach to arrive at the suita'le metrics is using the Moal)&uestion)Metric (echnique. As evident from the name, the M&M model consists of three layersN a Moal, a Set of &uestions, and lastly a Set of ?orresponding Metrics. .t is thus a hierarchical structure starting with a goal ,specifying purpose of measurement, o'%ect to 'e measured, issue to 'e measured, and viewpoint from which the measure is ta!en-. (he goal is refined into several questions that usually 'rea! down the issue into its ma%or components. "ach question is then refined into metrics, some of them o'%ective, some of them su'%ective. (he same metric can 'e used in order to answer different questions under the same goal. Several M&M models can also have questions and metrics in common, ma!ing sure that, when the measure is actually ta!en, the different viewpoints are ta!en into account correctly ,i.e., the metric might have different values when ta!en from different viewpoints-.

.n

order

to

give

an

e+ample

of

application

of

the

model:

7oal "urpose 5ssue &b$ect 1ie! "oint 8uestion Metric

.mprove the timeliness of ?hange /equest #rocessing from the #ro%ect Managers viewpoint hat is the current ?hange /equest #rocessing SpeedO Average ?ycle (ime Standard *eviation P cases outside of the upper limit .s the performance of the process improvingO ?urrent average cycle time <aseline average cycle time 3CC Q Su'%ective rating of managerRs satisfaction

8uestion Metric

When do you determine Metrics? hen the requirements are understood in a high)level, at this stage, the team si$e, pro%ect si$e must 'e !nown to an e+tent, in which the pro%ect is at a SdefinedS stage.