Sie sind auf Seite 1von 45

Measuring the Quality of

Standard Software:
the SAP Quality Index

Günther Limböck
Global Quality Governance
SAP AG, Walldorf

2. Workshop zur Software-Qualitätsmodellierung und -bewertung (SQMB '09)

Kaiserslautern, 3. März 2009


Agenda

Meine Geschichte bei SAP

Messen von Software Qualität: Herausforderungen

Das Projekt SAP Quality Index

Governance und Betrieb des SAP Quality Index

Erfahrungen und Ausblick

© SAP 2009 / Günther Limböck / Seite 2


SAP 1989: 1.400 Mitarbeiter
0,19 Mrd. EUR Umsatz
SAP R/2 4.3

© SAP 2009 / Günther Limböck / Seite 3


SAP 1998: 19.308 Mitarbeiter
4,33 Mrd. EUR Umsatz

SAP R/3 4.5

© SAP 2009 / Günther Limböck / Seite 4


SAP 2008: 52.000 Mitarbeiter
11,567 Mrd. Euro Umsatz
SAP Business Suite 7.0

© SAP 2009 / Günther Limböck / Seite 5


Agenda

Meine Geschichte bei SAP

Messen von Software Qualität: Herausforderungen

Das Projekt SAP Quality Index

Governance und Betrieb des SAP Quality Index

Erfahrungen und Ausblick

© SAP 2009 / Günther Limböck / Seite 6


Welche Aspekte von Software können wie gut
gemessen werden?

Welche Aussagen können üblicherweise


über Softwareprojekte gemacht werden?
Unser letztes großes Software
Entwicklungsprojekt
Fu
nk

hatte eine Dauer von 18 Monaten


tio
it

verursachte einen Aufwand von 80.000


Ze

na

Qualität Personentagen (80 Mio. Euro)


litä

hatte eine Funktionalität von 16.000 Functions


t

Points
hatte bei Auslieferung eine Qualität von 8,3 (auf
Aufwand / Kosten einer Scala von 1 bis 10)
hatte eine verbleibende Defect Density von 0,75
Defects pro Function Point

© SAP 2009 / Günther Limböck / Seite 7


Was soll man messen und was lieber nicht?

Zitate zu Messen
Messen? Nein! Messen? Ja!

„ All the statistics in the world can‘t measure


the warmth of a smile.
„ „In fact, the reduction – not necessarily the
elimination – of uncertainty is central to the
Chris Hart concept of measurement.”
Wikipedia

„ Intense love does not measure, it just gives.


Mother Teresa
„ If all guesses fail, try facts as an alternative.
Paul Strassman

„ Wer viel misst, misst Mist.


unbekannt
„ What gets measured gets managed

Peter Drucker

Messen soll verwendet werden, um Unsicherheit zu reduzieren und bei


Entscheidungen die Fakten zu kennen.
Messen soll auf Produkte/Prozesse angewendet werden, nicht auf Personen
© SAP 2008 / Page 8
Agenda

Meine Geschichte bei SAP

Messen von Software Qualität: Herausforderungen

Das Projekt SAP Quality Index

Governance und Betrieb des SAP Quality Index

Erfahrungen und Ausblick

© SAP 2009 / Günther Limböck / Seite 9


Eine Geschichte zum Messen von Qualität

In den 1980er-Jahren bestellte IBM Canada Ltd. in Markham, Ont.,


Bauteile von einem neuen japanischen Zulieferer

AUF
TR AG
… to
leri
werd ert
maxi en
mal 1
fehle ,5%
rhaft
Teile e

Begleitschreiben
Begleitschreiben
"Wir
"Wir wissen
wissen zwar
zwar nicht,
nicht, warum
warum
Sie
Sie 1,5
1,5 Prozent
Prozent fehlerhafte
fehlerhafte
Teile
Teile haben
haben möchten,
möchten,
aber
aber um Ihnen die Arbeit
um Ihnen die Arbeit zu
zu
erleichtern,
erleichtern, haben
haben wir
wir
sie
sie separat
separat verpackt."
verpackt."

© SAP 2009 / Günther Limböck / Seite 10


SAP Quality Index:
Der Projektauftrag im August 2005

Ein
Ein
gemeinsames
gemeinsames
Messsystem
Messsystem für
für
alle
alle
Entwicklungs-
Entwicklungs- Abdeckung
Abdeckung der
der
Anstoßen
Anstoßen von
von bereiche
bereiche wichtigsten
wichtigsten
Aktionen
Aktionen auf
auf Software-
Software-
Grundlage
Grundlage der
der Qualitätsaspekte
Qualitätsaspekte
Messergebnisse
Messergebnisse durch
durch Metriken
Metriken

Vergleichbarkeit
Vergleichbarkeit
Akzeptanz
Akzeptanz in der
der Werte
Werte für
für
den
den Produkte
Produkte
Entwicklungs-
Entwicklungs- unterschiedlicher
unterschiedlicher
bereichen
bereichen Art
Art und
und Größe
Größe

© SAP 2009 / Günther Limböck / Seite 11


Welche Daten werden für die Definition einer
KPI gebraucht?

Key Performance Indicator (KPI) Definition Template


Name: Type: Relevance to:
Product / Process / PIL phases
Goal: Project Invent / Define /
Objective / Subjective Develop / Deploy /
Direct / Indirect Optimize
Question:
SAP product standard
Breakdown granularity Significance
Metric description:
Project (10 = high, medium,
0 = low)
Application hierarchy
Others:

Measurement / Calculation procedure Implementation details

Aggregation rules Source (tool)


Tool changes
Measurement: Process changes
Start of measurement Business Information Development
Start of judgment, end of judgment
Maximum frequency

Comments

© SAP 2009 / Günther Limböck / Seite 12


Welche Aspekte sollen in einen Quality
Index?

ISO 9126 Qualitäts Modell

Wer?
Qualität aus Kundensicht Fokus auf externe Produkt-Qualität

© SAP 2009 / Günther Limböck / Seite 13


SAP Quality Index
Überblick über die Key Performance Indikatoren–

KPIs während der Development-Phase: Forecast der Produkt Qualität


Test of New Regression Test Performance after
Code Quality Limitations
Requirements Coverage Upgrade
Ensure formal code All new/enhanced Ensure stability of Stable performance of
quality by automated software requirements previously delivered Ensure functional
top X processes after
code checks of a program are tested product (code) with completeness,
upgrade precludes
successfully regression tests communicate
customers from
functional restrictions
needing new hardware
KPI: Code Quality

KPI Datasheet From Functional Correctness to Code Quality


Goal Regular
E nsure formal code quality by automated
code checks E xecution of
Code Checks
Answered Question
How many code check errors remain open Quality Index:
relatively to t he completed development Code Quality
effort?
of Program
Metric
K PI = Total number of open prio1/ 2 static SAP Standard Scope
Code Check messages from all Code Check for Code Checks
f rameworks per person days development
effort measured in 1/1000 incl. Priorities
Measurement/Judgmen t defined by
P 2D D2P RTC + 12 mont hs Code Check Meeting

Product Standard
Functional Correctness
no Code Check errors of
priority 1 or 2

© SA P 20 08 / SAP Q u alit y In de x 2 0 08 ed itio n / Pa ge 1 6

KPIs nach Auslieferung: KPIs: Produkt Qualität aus Kundensicht


Aggregated
Ramp-Up Survey Bug Solving Notes Usability
Validation Results
Ensure Product Get comprehensive Keep the number of Achieve high-quality
Standard Compliance understanding of corrections after user experiences with
by the very first customer’s satisfaction shipment as low as SAP products for our
customer with SAP solutions possible customers
during Ramp-Up
KPI: Aggregated Validation Results KPI: Ramp-Up Satisfaction Survey KPI: Bug-Solving Notes

KPI Datasheet Overview on KPI Calculation and Aggregation KPI Datasheet KPI Datasheet Overview on KPI Calculation and Aggregation
Goal PTU A
Dev .
efforts
Validation res ults Goal Goal
Measure and provide t ransparency on Get a comprehensive understanding of the Provide transparency about the quality
Progr am A1
P roduct Standard Compliance before customer’s satisfaction with the SAP solutions perceived by our customers for the SAP
shipment Progr am A2
during Ramp-Up product s after they have been released to
customers
Answered Question Progr am A3 Answered Question
Degree of compliance with validated product How does t he Ramp-Up customers rat e the Answered Question
standards. Calculati on: quality, document ation, cost-performance Do shipped products have defects
Do products fulfill the requirement s of Product ration etc. of our solutions? Different calculation rul es on program and on P TU l evel
Calculation on program level over all available average validation encountered by customers? How many
S tandards f or shipment ? customers are impacted due t o those defects Program l evel:
results (calculation model: each RE D=0 pts; each Y ELLOW=5 pts;
each GREEN=10pt s).
Metric and hence af fecting SAP’s reliability? St atic indexation rule of 0-10 points
Metric Ramp-Up customer satisfaction survey = Three quality indicator zones, Red, Yellow and Green at program
Average of 8 product relat ed questions Metric level
AvgValiRes ult
# RED * 0 # YELLOW * 5 # GREEN *10 Additional remarks:
# RED # YELLOW # GREEN Number of Bug Solving Notes reported in a
Validation results of all validation projects are aggregat ed, Measurement/Judgment mont h per 1000 days of Dev Eff ort for t hat Aggregation on P TU l evel:
Measurement/Judgmen t i. e. the result of a re-validation or SP -validation does not overwrite Program; e.g. Metric = (Number of Bug
the result of the original release validation P2R RTC + 24 months Weighting f actor: A mount of development effort
RTC RTC + 12 months Solving Notes / Tot al Dev Eff ort) * 1.000
Using this normalization factor leads to a dynamic indexation
Only the product related questi ons of the ramp-up satisfaction Measurement/Judgment from 0 to 10 pts and provides comparability and distinction
survey are taken into account, for example: P2R RTC + 36 months between programs of diff erent size.
Q06: How would you rat e your overall satisfact ion with S AP Aggregation through weighted evaluation of t he K PI values on
[solution]? program level provides the PTU values.
Q07: How would you rat e the quality and reliabilit y of this SAP
[solution]?
Q09: How would you rat e th e usabilit y and the ease of learning of
this SAP [ solution]?
Q14: How would you rat e the cost-performance ratio of the
soft ware?

© SA P 20 08 / SAP Q u alit y In de x 2 0 08 ed itio n / Pa ge 1 7 © SA P 20 08 / SAP Q ua lity In de x 2 00 8 ed itio n / Pa ge 1 8 © SA P 20 08 / SAP Q ua lity In de x 2 00 8 ed itio n / Pa ge 1 9

© SAP 2009 / Günther Limböck / Seite 14


Beschreibung der KPI:
Test of New Requirements Beis
piel

KPI Datasheet: Overview on KPI Calculation and Aggregation


Goal Project Test Plan
All new software requirements of a
program are tested successfully
Specification 1 Test case 1

Answered Question
Test case 2
How many software requirements are Requirement 1
covered by test cases and tested
successfully? Requirement 2 Test case 3

Metric Requirement 3 Test case 4

# Re q. Green test cases for Req. i


KPI = # test cases for Req. i * # Req.
Requirement Repository Test Workbench
i =1
Test Result
Measurement/judgment Requirement 1 = 0.5
P2D RTC + 12 months Requirement 2 = 0
Requirement 3 = 0.67

0 .5 0 0 . 67
Result 0 .39
3

© SAP 2009 / Günther Limböck / Seite 15


SAP Quality Index
Überblick über die Key Performance Indikatoren–

KPIs während der Development-Phase: Forecast der Produkt Qualität


Test of New Regression Test Performance after
Code Quality Limitations
Requirements Coverage Upgrade
Ensure formal code All new/enhanced Ensure stability of Stable performance of
quality by automated software requirements previously delivered Ensure functional
top X processes after
code checks of a program are tested product (code) with completeness,
upgrade precludes
successfully regression tests communicate
customers from
functional restrictions
needing new hardware
KPI: Code Quality

KPI Datasheet From Functional Correctness to Code Quality


Goal Regular
E nsure formal code quality by automated
code checks E xecution of
Code Checks
Answered Question
How many code check errors remain open Quality Index:
relatively to t he completed development Code Quality
effort?
of Program
Metric
K PI = Total number of open prio1/ 2 static SAP Standard Scope
Code Check messages from all Code Check for Code Checks
f rameworks per person days development
effort measured in 1/1000 incl. Priorities
Measurement/Judgmen t defined by
P 2D D2P RTC + 12 mont hs Code Check Meeting

Product Standard
Functional Correctness
no Code Check errors of
priority 1 or 2

© SA P 20 08 / SAP Q u alit y In de x 2 0 08 ed itio n / Pa ge 1 6

KPIs nach Auslieferung: KPIs: Produkt Qualität aus Kundensicht


Aggregated
Ramp-Up Survey Bug Solving Notes Usability
Validation Results
Ensure Product Get comprehensive Keep the number of Achieve high-quality
Standard Compliance understanding of corrections after user experiences with
by the very first customer’s satisfaction shipment as low as SAP products for our
customer with SAP solutions possible customers
during Ramp-Up
KPI: Aggregated Validation Results KPI: Ramp-Up Satisfaction Survey KPI: Bug-Solving Notes

KPI Datasheet Overview on KPI Calculation and Aggregation KPI Datasheet KPI Datasheet Overview on KPI Calculation and Aggregation
Goal PTU A
Dev .
efforts
Validation res ults Goal Goal
Measure and provide t ransparency on Get a comprehensive understanding of the Provide transparency about the quality
Progr am A1
P roduct Standard Compliance before customer’s satisfaction with the SAP solutions perceived by our customers for the SAP
shipment Progr am A2
during Ramp-Up product s after they have been released to
customers
Answered Question Progr am A3 Answered Question
Degree of compliance with validated product How does t he Ramp-Up customers rat e the Answered Question
standards. Calculati on: quality, document ation, cost-performance Do shipped products have defects
Do products fulfill the requirement s of Product ration etc. of our solutions? Different calculation rul es on program and on P TU l evel
Calculation on program level over all available average validation encountered by customers? How many
S tandards f or shipment ? customers are impacted due t o those defects Program l evel:
results (calculation model: each RE D=0 pts; each Y ELLOW=5 pts;
each GREEN=10pt s).
Metric and hence af fecting SAP’s reliability? St atic indexation rule of 0-10 points
Metric Ramp-Up customer satisfaction survey = Three quality indicator zones, Red, Yellow and Green at program
Average of 8 product relat ed questions Metric level
AvgValiRes ult
# RED * 0 # YELLOW * 5 # GREEN *10 Additional remarks:
# RED # YELLOW # GREEN Number of Bug Solving Notes reported in a
Validation results of all validation projects are aggregat ed, Measurement/Judgment mont h per 1000 days of Dev Eff ort for t hat Aggregation on P TU l evel:
Measurement/Judgmen t i. e. the result of a re-validation or SP -validation does not overwrite Program; e.g. Metric = (Number of Bug
the result of the original release validation P2R RTC + 24 months Weighting f actor: A mount of development effort
RTC RTC + 12 months Solving Notes / Tot al Dev Eff ort) * 1.000
Using this normalization factor leads to a dynamic indexation
Only the product related questi ons of the ramp-up satisfaction Measurement/Judgment from 0 to 10 pts and provides comparability and distinction
survey are taken into account, for example: P2R RTC + 36 months between programs of diff erent size.
Q06: How would you rat e your overall satisfact ion with S AP Aggregation through weighted evaluation of t he K PI values on
[solution]? program level provides the PTU values.
Q07: How would you rat e the quality and reliabilit y of this SAP
[solution]?
Q09: How would you rat e th e usabilit y and the ease of learning of
this SAP [ solution]?
Q14: How would you rat e the cost-performance ratio of the
soft ware?

© SA P 20 08 / SAP Q u alit y In de x 2 0 08 ed itio n / Pa ge 1 7 © SA P 20 08 / SAP Q ua lity In de x 2 00 8 ed itio n / Pa ge 1 8 © SA P 20 08 / SAP Q ua lity In de x 2 00 8 ed itio n / Pa ge 1 9

© SAP 2009 / Günther Limböck / Seite 16


Für welche Entität soll Qualität gemessen
werden?
Organisationsstruktur der SAP-Entwicklung

SAP-Produkt-
entwicklung

PTU = Product
Technology Unit PTU 1 PTU 2 PTU 3

Programm Programm Programm Programm =


SCM 5.0 SCM 6.0 SCM 7.0 Produktversion

Programme repräsentieren das, was der Kunde auf der DVD bekommt
Ein Programm ist daher die zentrale Entität, auf der Qualität von
Software bei SAP gemessen wird

© SAP 2009 / Günther Limböck / Seite 17


Wann soll gemessen werden, wann soll
bewertet?
Zeitachse pro Programm:

12 Monate
Programm XYZ
KPIs vor der Messung Bewertung
Auslieferung

KPIs nach der


Bewertung
Auslieferung

P2D D2P RTC

P2D = Q-Gate D2P = Q-Gate RTC =


Planning to Development to Release to
Development Production Customer
Beginn der Ende der
Beginn der
Bewertungs- Bewertungs-
Messphase
phase phase

In der ersten Version gab es “exaktere” Meilensteine, wie z.B. „End of Test“
Erfahrung nach einem Jahr: einfach ist besser!
© SAP 2009 / Günther Limböck / Seite 18
Bewertung jeder KPI und
Zuordnung von KPI-Werten zur Index-Skala

Für alle KPIs wurden Regeln definiert, die die KPI–spezifischen Werte einer
gemeinsamen Q–Index-Skala von 0 bis 10 zuordnet

Green
KPI-Wertebereiche
Q-Index-Bereiche
KPI–spezifische Yellow
Für alle KPIs identisch
Bewertung, z.B.:

0 % < x 80% rot


0 x<5 rot
80 % < x 90% gelb
5 x<7 gelb
90 % < x 100% grün
7 x 10 grün

© SAP 2009 / Günther Limböck / Seite 19


Aggregationsregeln zur Berechnung der KPI-
Werte auf verschiedenen Ebenen
Für alle KPIs gilt der Entwicklungsaufwand als Gewichtungsfaktor. Alle KPIs
werden im Index gleich gewichtet.
Beispiel einer Aggregation für eine KPI:
Entwicklungs- Gewicht- Gewichte-
PTU Program Index-Wert
aufwand ung ter Index Q-Index-
PTU 1 Program 1 4.000 80,00% 6,0 4,8
Wert PTU 1

PTU 1 Program 2 1.000 20,00% 8,0 1,6

Summe PTU 1 5.000 100,00% 6,40


Q-Index-
PTU 2 Program 3 20.000 16,67% 6.0 1,0 Wert PTU 2
PTU 2 Program 4 100.000 83,33% 8.0 6,67

Summe PTU 2 120.000 100,00% 7,67


SAP-Q-
SAP Total 125.000 7,62 Index-Wert

In der ersten Version gab es “exaktere” Gewichtungsfaktoren


Erfahrung nach einem Jahr: einfach ist besser!

© SAP 2009 / Günther Limböck / Seite 20


SAP Quality Index User Interface –
Sicht SAP gesamt
Beis
piel

© SAP 2009 / Günther Limböck / Seite 21


SAP Quality Index User Interface –
Sicht je PTU
Beis
piel

© SAP 2009 / Günther Limböck / Seite 22


SAP Quality Index User Interface –
Sicht je Programm
Beis
piel

© SAP 2009 / Günther Limböck / Seite 23


Agenda

Meine Geschichte bei SAP

Messen von Software Qualität: Herausforderungen

Das Projekt SAP Quality Index

Governance und Betrieb des SAP Quality Index

Erfahrungen und Ausblick

© SAP 2009 / Günther Limböck / Seite 24


SAP „Quality Goal Compliance Declaration“
für jedes Programm
Beis
piel

© SAP 2009 / Günther Limböck / Seite 25


Q-Index-Betrieb: Vorläufiger und endgültiger
Bericht

Q-Index-Datensammlung

Abstimmung
Monatlicher Q- mit den QM-Heads der
Endgültiger Q-Index
Report Index- Entwicklungsbereiche
Berichtszyklus und Beschluss von
Maßnahmen

Vorläufiger
Q-Index Report

© SAP 2009 / Günther Limböck / Seite 26


SAP Quality Index: Zusammenfassung

Ein
Ein
gemeinsames
gemeinsames
Messsystem
Messsystem für
für
alle
alle
Entwicklungs-
Entwicklungs- Abdeckung
Abdeckung der
der
Anstoßen
Anstoßen von
von bereiche
bereiche wichtigsten
wichtigsten
Aktionen
Aktionen auf
auf Software-
Software-
Grundlage
Grundlage der
der Qualitätsaspekte
Qualitätsaspekte
Messergebnisse
Messergebnisse durch
durch Metriken
Metriken

Vergleichbarkeit
Vergleichbarkeit
Akzeptanz
Akzeptanz in der
der Werte
Werte für
für
den
den Produkte
Produkte
Entwicklungs-
Entwicklungs- unterschiedlicher
unterschiedlicher
bereichen
bereichen Art
Art und
und Größe
Größe

© SAP 2009 / Günther Limböck / Seite 27


Agenda

Meine Geschichte bei SAP

Messen von Software Qualität: Herausforderungen

Das Projekt SAP Quality Index

Governance und Betrieb des SAP Quality Index

Erfahrungen und Ausblick

© SAP 2009 / Günther Limböck / Seite 28


Erfahrungen – 1 –

KPI-Definition und Indexberechnung


Goal-Question-Metric-Methode ist sehr
nützlich bei der KPI-Definition
Aggregationsregeln und Zeitachsen:
einfach ist besser!

Reporting
Erwartungen an Steigerung der
Transparenz wurden übertroffen
Wichtigster Erfolgsfaktor: Sichtbarkeit
auf Vorstandsebene
Hohe Datenqualität ist erforderlich
Das UI ist wichtig für die Akzeptanz
(vor allem im Senior Management)

© SAP 2009 / Günther Limböck / Seite 29


Erfahrungen – 2 –

Governance und Betrieb


Reports ohne Maßnahmen sind
Datenfriedhöfe

Regelmäßige Überprüfung der KPI-


Definitionen ist notwendig, um sicher zu
stellen, dass die Qualität optimiert wird und
nicht der Index

Betrieb des Index erfordert hohe


Qualifikation

Betrieb des Index erfordert hohe


Kommunikationsfähigkeit

Betrieb des Q-Index benötigt ein


starkes Governance Team

© SAP 2009 / Günther Limböck / Seite 30


Ausblick

Systematische Datenanalyse
basierend auf den Q-Index Daten
zu einem frühen Zeitpunkt
Erhöhung der Vorhersagbarkeit
von Qualität
Integration von Akquisitionen
(z.B. SAP Business Objects)
Unterstützung von Technologien
neben ABAP und JAVA

© SAP 2009 / Günther Limböck / Seite 31


Danke für Ihre Aufmerksamkeit!

© SAP 2009 / Günther Limböck / Seite 32


Anhang

© SAP 2009 / Günther Limböck / Seite 33


SAP Quality Index KPIs: Kunden-Feedback

Feedback bzgl. der Relevanz der KPIs durch eine SAP-Kundengruppe

5,00
4,50
4,00
3,50
3,00
2,50
2,00
1,50
1,00
0,50
0,00
Test of New Regression Performance Code Usability Bug Aggregated Ramp-Up Limitations Resource
Requirements Test after Quality Solving Validation Survey Commitment
Coverage Upgrade Notes Results Rate

Die KPIs sind dazu geeignet, die Software-Produktqualität zu messen


Die KPI Resource Commitment Rate wurde 2008 aus dem SAP Quality Index gestrichen

© SAP 2008 / Page 34


SAP aktuell

Jahresumsatz der SAP AG 2007: 10,25 Mrd. Euro


Mehr als 76.000 Unternehmen setzen SAP ein
Mehr als 25 Branchenlösungen
51.863 SAP-Mitarbeiter (Stand: September 2008)
12 Millionen Anwender in über 120 Ländern nutzen
SAP-Lösungen zur
Integration von Geschäftsprozessen
Stärkung der Wettbewerbsfähigkeit
schnelleren Rendite bei niedrigeren Systemgesamtkosten
Einzigartiges Partnernetzwerk
Mehr als 1,3 Millionen Community Mitglieder (SDN und BPX)
Mehr als 1.800 zertifizierte Partnerlösungen für SAP NetWeaver
15 Industry Value Networks
© SAP 2009 / Günther Limböck / Seite 35
SAPs Forschungs- und Entwicklungsorganisation:
ca. 15.000 Entwickler an 8 Entwicklungsstandorten

Deutschland
Walldorf
Engineering
betriebswirtsch.
Know-how Ungarn
Kanada Budapest
Montreal
Bulgarien
Sofia China
Shanghai
USA
Palo Alto Israel
Betriebswirtsch. Ra’anana
Innovation Technologische Indien
Ökosystem Innovation Bangalore
Talent-Pool

Alle Standorte arbeiten gemäß einem Qualitätsmanagement-System


(ISO 9001-zertifiziert)

© SAP 2009 / Günther Limböck / Seite 36


KPI: Code Quality Beis
piel

KPI Datasheet From Functional Correctness to Code Quality

Goal Regular
Ensure formal code quality by Execution of
automated code checks
Code Checks
Answered Question Quality Index:
How many code check errors remain Code Quality
open relatively to the completed of Program
development effort?
SAP Standard Scope
Metric
for Code Checks
KPI = Total number of open prio1/2
static Code Check messages from all incl. Priorities
Code Check frameworks per person defined by
days development effort measured in Code Check Meeting
1/1000

Measurement/Judgment
P2D D2P RTC + 12 months Product Standard
Functional Correctness
no Code Check errors of
priority 1 or 2

© SAP 2009 / Günther Limböck / Seite 37


KPI: Test of New Requirements Beis
piel

KPI Datasheet Overview on KPI Calculation and Aggregation


Goal Project Test Plan
All new software requirements of a
program are tested successfully
Specification 1 Test case 1

Answered Question
Test case 2
How many software requirements are Requirement 1
covered by test cases and tested
successfully? Requirement 2 Test case 3

Metric Requirement 3 Test case 4

# Re q. Green test cases for Req. i Requirement Repository Test Workbench


KPI = # test cases for Req. i * # Req.
i =1
Test Result
Measurement/judgment Requirement 1 = 0.5
P2D RTC + 12 months Requirement 2 = 0
Requirement 3 = 0.67

0 .5 0 0 . 67
Result 0 .39
3

© SAP 2009 / Günther Limböck / Seite 38


KPI: Regression Test Coverage Beis
piel

KPI Datasheet Overview on KPI Calculation and Aggregation


Goal Top X Scenarios Test Plan
Ensure stability of previously delivered
product (code) with regression tests
Process 1 Test case 1

Answered Question
Test case 2
Are regression tests successfully Process step 1
running for the top X processes,
process steps? Process step 2 Test case 3

Metric Process step 3 Test case 4

# Pr .step Green test cases Proc.step i


KPI # Test cases Proc.step i * # Proc.steps Test Object Repository Test Workbench
i 1

Measurement/Judgment Test Result


P2D RTC + 12 months Requirement 1 = 0.5
Requirement 2 = 0
Requirement 3 = 0.67

0 .5 0 0 . 67
Result 0 .39
3

© SAP 2009 / Günther Limböck / Seite 39


KPI: Limitations Beis
piel

KPI Datasheet Overview on KPI Calculation and Aggregation


Goal
Provide transparency about functional
restrictions

Answered Question
Do shipped products have functional
restrictions?
Do these functional restrictions have an
impact on SAPs reliability?

Metric
Prio-A limitations are the leading metric
factors. No Prio-A limitations allowed at all.

Measurement/Judgment
RTC RTC + 12 months

© SAP 2009 / Günther Limböck / Seite 40


KPI: Aggregated Validation Results Beis
piel

KPI Datasheet Overview on KPI Calculation and Aggregation


Dev.
Goal PTU A efforts
Validation results

Measure and provide transparency on Program A1


Product Standard Compliance before
Program A2
shipment
Program A3
Answered Question
Degree of compliance with validated Calculation:
product standards. Calculation on program level over all available average validation
Do products fulfill the requirements of results (calculation model: each RED=0 pts; each YELLOW=5 pts;
Product Standards for shipment? each GREEN=10pts).

Metric Additional remarks:


Validation results of all validation projects are aggregated,
# RED *0 #YELLOW *5 #GREEN *10 i.e. the result of a re-validation or SP-validation does not overwrite
AvgValiRes ult # RED #YELLOW #GREEN the result of the original release validation

Measurement/Judgment
RTC RTC + 12 months

© SAP 2009 / Günther Limböck / Seite 41


KPI: Performance After Upgrade
Beis
piel

KPI Datasheet Overview on KPI Calculation and Aggregation


Goal
Ensure stable performance of top X
scenarios, processes, and process
steps after upgrade, so customers
shouldn’t need new hardware.

Answered Question
How is the performance compared to
last release?

Metric
Index will be calculated for a program
out of the four Performance indicators:
Data Base Load, Memory
Consumption, CPU Consumption,
Network Load.

Aggregation on Program level:


Measurement/Judgment Calculate the mean value of all four performance indicators for each top x
process; then calculate the performance after upgrade KPI for the whole
P2D RTC + 12 months program as a mean value of those top x processes.

© SAP 2009 / Günther Limböck / Seite 42


KPI: Ramp-Up Satisfaction Survey Beis
piel

KPI Datasheet
Goal
Get a comprehensive understanding of
the customer’s satisfaction with the
SAP solutions during Ramp-Up

Answered Question
How does the Ramp-Up customers
rate the quality, documentation, cost-
performance ration etc. of our
solutions?

Metric
Ramp-Up customer satisfaction survey
= Average of 8 product related
questions
Only the product related questions of the ramp-up satisfaction
Measurement/Judgment survey are taken into account, for example:
P2R RTC + 24 months Q06: How would you rate your overall satisfaction with SAP
[solution]?
Q07: How would you rate the quality and reliability of this SAP
[solution]?
Q09: How would you rate the usability and the ease of learning of
this SAP [solution]?
Q14: How would you rate the cost-performance ratio of the
software?

© SAP 2009 / Günther Limböck / Seite 43


KPI: Bug-Solving Notes Beis
piel

KPI Datasheet Overview on KPI Calculation and Aggregation


Goal
Provide transparency about the quality
perceived by our customers for the
SAP products after they have been
released to customers

Answered Question Different calculation rules on program and on PTU level


Do shipped products have defects
encountered by customers? How many Program level:
Static indexation rule of 0-10 points
customers are impacted due to those Three quality indicator zones, Red, Yellow and Green at program
defects and hence affecting SAP’s level
reliability?
Aggregation on PTU level:
Metric Weighting factor: Amount of development effort
Number of Bug Solving Notes reported Using this normalization factor leads to a dynamic indexation
in a month per 1000 days of Dev Effort from 0 to 10 pts and provides comparability and distinction
for that Program; e.g. Metric = (Number between programs of different size.
of Bug Solving Notes / Total Dev Effort)
* 1.000 Aggregation through weighted evaluation of the KPI values on
program level provides the PTU values.
Measurement/Judgment
P2R RTC + 36 months

© SAP 2009 / Günther Limböck / Seite 44


KPI: Usability Beis
piel

KPI Datasheet Overview on KPI Calculation and Aggregation


Goal
Achieve high-quality user experiences
with SAP products for our customers

Answered Question
How well do targeted end-users
perform representative tasks, in terms
of task-completion rates and times, and
how satisfied are they with the
applications?

Metric
2 * U ser Ef f ectiv en ess + U ser Ef f icien cy + U ser S atisf acti on
K PI = 4
User Effectiven ess = Time - limited, task - completion rate
User Efficiency = Task - completion time
User Satisfacti on = Magnitude Estimation Technique (MET)

Measurement/Judgment
At intervals that correspond to real
application lifecycles. Benchmark tests
on only a subset of applications. Remark: Definition of user effectiveness, user efficiency, and
RTC RTC + 12 months user satisfaction according to ISO 9241

© SAP 2009 / Günther Limböck / Seite 45