NAVAIR SW Estimation

Improvement Program
Rick Holcomb
AIR-4.2 Cost Department
(301) 342-2450
holcombra@navair.navy.mil

30 January, 2003

Dan Galorath
Galorath Incorporated
(310) 414-3222
dgalorath@galorath.com

36th ADoDCAS

Agenda

Whats Necessary for Accurate Software

Estimation?
NAVAIR Initiatives
Galoraths Phase II Implementation of
SBIR N01-020

30 January, 2003

36th ADoDCAS

Whats Necessary
for Accurate SW
Estimation?
30 January, 2003

36th ADoDCAS

What is Accuracy?
How close is the initial estimate to the
final estimate?
How has functionality changed from
the initial technical baseline to the
final product? What impact has this
had on the estimate?

30 January, 2003

36th ADoDCAS

Teamwork & Discipline are

Required
Improvement in SW Estimation requires
participation by both NAVAIR & its Industry
Partners, not just the Cost Department.
Requires mature, disciplined software processes
for all phases of the life-cycle for both the
Acquisition and Development organizations to
produce an accurate complete Technical &
Programmatic Baseline.
the major factor in successful software development is
disciplined execution. Report of the Defense Science Board
Taskforce on Defense Software, November 2000.

30 January, 2003

36th ADoDCAS

A Mature Disciplined Development

Environment is Important
Generally accurate software estimates can only be produced
when high quality data about the development
organization(s) is available.
High quality software data can only be generated by mature,
disciplined Acquisition and Development organizations.
Even if data is available for an immature, undisciplined
organization, attempting to estimate for this organization is
problematic.
Having good data on an immature undisciplined organization is an
oxymoron.
Once data starts to become available on an organization, it indicates
at least the start of an effort to increase the maturity and discipline of
the software development process.

30 January, 2003

36th ADoDCAS

SW-CMM Summary
Levels of Process Maturity
5

RISK

LOW

HIGH 1
30 January, 2003

4
3
2

MANAGED

DEFINED

REPEATABLE
INITIAL

Continuously
improving process

OPTIMIZING

Predictable process
Standard, consistent process

Disciplined
Ad hoc; occasionally chaotic

36th ADoDCAS

Probability of Success
120%

Product Succe ss

100%
80%
60%
SEI 1 & 2 Success

40%

SEI 3,4, & 5 Success

20%
0%
0

10

100

1,000

10,000

100,000

SW Size (Function Points)

(1) Capers Jones, Becoming Best In Class, Software Productivity Research, 1995 briefing
(2) 1 Function Point = 50 SLOC ADA, 40 SLOC C++, 320 SLOC Assembly

30 January, 2003

36th ADoDCAS

With & Without Accurate

Data
(Efforts = Labor Hours)

Over/Under Percentage

140%

0%

-140%

. .
.. .
..... .... ..... ..... ................. ....
.. ...... ......... .. . .
.. . .
.
.
.
.
.
.
.
.
. . ... ..
. . . . . .. .. .. .. .. . . . .. .. . .
.
.. .
..
. .. . . . . . .. .
. .. .. . .. . .. . ... . . .. .. .
. . .. ... . .. . .. .
.. . . . .. . .. . . ....
. . .. . . . . .. . . . .
. .. .
. ..
.
.
.

Without Historical Data

Variance between + 20% to - 145%
(Mostly Level 1 & 2)
John Vu, Boeing, keynote talk at SEPG 97,
Software Process Improvement Journey (From Level 1 to Level 5)

30 January, 2003

With Historical Data

Variance between - 20% to + 20%
(Level 3)
(Based on 120 projects in
Boeing Information Systems)

36th ADoDCAS

NAVAIR
Initiatives

30 January, 2003

36th ADoDCAS

10

Process Improvement
Developers of ACAT I, II, III, & IV Software
Intensive Systems will achieve a SEI SW-CMM
Level 3 or equivalent.
NAVAIRINST 5234.1 Policy on Software Evaluations for Naval Air Systems Command Programs

Achieve a corporate SW-CMM or CMMI Level III by

2005, and Level V by 2008.
NAVAIRINST 5234.2 Requirements for Process Improvement Actions for Naval Air Systems Command
Software Acquisition, Development and Life-Cycle Support

Implement Corporate Requirements for Software

Metrics & Measurement Collection.
NAVAIRINST 5234.5 Naval Air Systems Command Metrics for Software Intensive Systems

30 January, 2003

36th ADoDCAS

11

Software Resource Center

The NAVAIR Software Resource Center provides expert
training & assistance to programs involved in the
development of software intensive systems in areas such
as:
SW-CMM & CMMI
Team Software Process (TSP) / Personnel Software Process (PSP)
Practical Software & Systems Measurement (PSM)
Independent Expert Program Reviews (IEPR)
Software Estimation
Software Engineering Acquisition, Management & Development
Lessons Learned

30 January, 2003

36th ADoDCAS

12

Communications
The best results are achieved
when all levels of the organization
agree with the need for change
rather then having it forced upon
them.
A major aspect of NAVAIRs
initiatives is the training and
communications with the
workforce.

30 January, 2003

36th ADoDCAS

13

Progress
F/A-18 Weapons Software Support Activity (WSSA), SWCMM Level V.
Longest history of process improvement in NAVAIR. Prime
contractor, Boeing, is also a Level IV. Recent developments have
been very successful at meeting cost and schedule objectives.
One of Top 5 DoD Software projects in 2001,
http://www.stsc.hill.af.mil/top5projects/

AV8B WSSA, SW-SMM Level IV and TSP/PSP adopted.

AV-8Bs Experience Using the TSP to Accelerate SW-CMM Adoption,
Crosstalk Sep 2002,
http://www.stsc.hill.af.mil/crosstalk/2002/09/hefley.html

1 SW-CMM Level III Group

3 SW-CMM Level II Groups
10 Groups adopting TSP/PSP
18 Groups started SW-CMM Process Improvement Efforts
8 Groups investigating CMMI

30 January, 2003

36th ADoDCAS

14

Improved Tools
SBIR N01-020, Software Cost & Schedule
Estimation:
Demonstrate processes and tools that will estimate software
cost and schedule with a standard deviation of 10% or less.
Make best possible use of limited data sample sets for
developing software cost and schedule estimates.
Enhance update of estimates based on actual project data.
Enhance collection and archival of project software metrics
and measurements in order to support calibration and improve
accuracy for both current and future projects

30 January, 2003

36th ADoDCAS

15

Galoraths Phase II
Implementation of
SBIR N01-020

30 January, 2003

36th ADoDCAS

16

Project Plan

ID
1
2
106
160
161
246
273
300

Task Name

ProjectMiner
ProjectMiner Release 1
ProjectMiner Release 1B
SEER-SEM Integration
ProjectMiner Release 2
ProjectMiner Release 3
ProjectMiner Release 4
Transition - non-iterated tasks

30 January, 2003

Finish

anuary 21
June 1
2/10
4/14
6/16

8/18

October 11
February 21
10/20 12/22
2/23
4/27

July 1
6/29

8/31

Nov ember 11
11/2
1/4

March
3/7

Mon 4/19/04
Thu 7/3/03
Tue 9/23/03
Tue 10/21/03
Mon 1/26/04
Wed 2/18/04
Tue 3/23/04
Mon 4/19/04

36th ADoDCAS

17

Data Rights
Government has standard SBIR data rights
for application.
Can view all requirements, design and source code
of product.
After 5 years, Government may go to another
source to modify application, IF:
9Galorath Incorporated has not copyrighted the application;
however, it probably will.
9Government has not contracted with Galorath for further
modifications of the application.

30 January, 2003

36th ADoDCAS

18

Plan to Meet NAVAIR Goals

Sizing
Sizing made easier, and more possible earlier in the estimation process

Estimating
Decision makers will better understand the origins of an estimate
Estimate will be more accurate, more reliable, and better integrated into the
project life cycle
Impact of new technologies and specific circumstances will be captured

Planning
Estimate will be integrated into earned value and other project management
methods
An integrated estimating methodology will be supported

Data Warehousing
Past project acutals will be sorted and accessed for more accurate, more
defensible estimates

30 January, 2003

36th ADoDCAS

19

Accurate Estimating Requires

Attention to Sizing
Size by Comparison
Highly accurate
method of sizing new
pieces of software
against known past
items.

COTS Estimating By Characteristic

Analogies
Size new jobs against
standardized benchmarks.

30 January, 2003

+ Characteristic
Rating
- Rigor of requirements
Very High
- Number of interfaces
70-100
- Complexity
Very High
COTS Estimating
COTS Characteristic
By Characteristic
- Application Domain Specify
Aero
- Application Type
Signal
processing
Characteristic
Rating
- << Next characteristic >>
Rigor of requirements
High

COTS entry

Help

Next >>
OK

Cancel

Sizing based on characteristics, a completely different

approach to scope.
36th ADoDCAS

20

Detailed Sizing Workups

SizingEstimating
View
COTS
By Characteristic
+ SIZE
- Component A
- Component B - part of A
- Component C
- Component D
- Component E
- Component F - part of A
- Component G
- Component H
- Component I
- Component J - part of A
- Component K
- Component L

Metric

Quantity

Legacy

SLOC
SLOC
ELOC
Classes
SLOC
Use Cases
COTS Sizing
SLOC
SLOC
Function Based
Detailed Objects
SLOC

1700
120
450
12...
60
2...
select for info
15000
300
select for info
45...
1200

Pre-existing
Pre-existing
Pre-existing
New
New
Pre-existing
COTS
Pre-existing
Pre-existing
Pre-existing
Pre-existing
Pre-existing

Rather than one or two major entries,

many separate sizing sub-entries can be
made within any particular program
being estimated. A wide range of sizing
methods can be combined or used
alongside each other.
Result
Estimates can be top-down (general)
OR bottom-up (detailed).
30 January, 2003

36th ADoDCAS

21

SEER NexGens ProjectMiner

Preliminary Proven Performance

Using extremely
limited, relevant
past data.
Results obtained using 5
aviation data points and
Least Absolute Error
estimation.

Using sufficient
relevant past data.

Within 20%

Within 12%

Within
10%

Using extremely
limited, relevant
past data, with good
assumptions about
predicted project.
Results obtained using
maximum information pair-wise
comparisons, with accurate
conjecture on probable range.

Results obtained using

neural network.

With NO relevant past data.

Results regularly obtained using SEER-SEM,
in the future also using ProjectMiner
consensus estimates coupled with the
Development Factors model.

With a modest amount

of NAVAIR data.
Using a preliminary version of ProjectMiner,
mixing Least Absolute Error, Weighted
Least Squares and Maximum Likelihood .

results on data that would be used in a production system. All data and tests are available for independent review.
* These are actual
Please note that final results should be better, as the ProjectMiner methodology is still being prototyped.

30 January, 2003

36th ADoDCAS

22

Best
Models,
Methods
& Data

Settings
For
Current
Estimate

Estimating Methods

D
at
a

NAVAIR,
contractors,
industry,
etc

Se
t

ProjectMiner Concept
Many Approaches Make a Better
Estimate

Multiple
statistical
methods,
etc
30 January, 2003

Combined
Combined
Results
Resultsof
of
Various
Various
Statistical
Statistical
Methods
Methods

Functional Forms

y=a*xb, etc
36th ADoDCAS

23

ProjectMiner How It Works

Past
Project Data

Determine
Relevant
Data

Unlimited number
of core models

30 January, 2003

Develop
Potentially
Relevant
Equations

Apply Data and

Equations To
Estimating
Methods

Adjust Best
Estimate With
Empirically Derived
Add-On Models

Intelligently
Combine
Best
Approaches

Unlimited
number of addon models

Assess
Performance
Of Each
Approach

36th ADoDCAS

24

Key Leveraging Technologies:

Voting and Conformal Estimating
Voting
Votingfinds
findsthe
themost
most
accurate
accurateestimating
estimating
approaches.
approaches.

n
ion riso r
s
res mpa othe
g
re co

NAVAIR data
Contractor data

Conformal
Conformalestimating
estimatingkeeps
keeps
unique
uniqueinfo
infoabout
aboutpast
pastprojects
projects
and
andthen
thenuses
usesthat
thatto
toimprove
improve
future
futureestimates.
estimates.
Standard
Info:

Unique
Info:

Size, Platform,
Application, etc.

Specific Tools,
Circumstances, etc.

Other data

Combines the best methods, paying

particular attention to:
The most relevant data sources.
Embedded intelligence about performance
of specific methods and models against
specific data.
30 January, 2003

Invaluable additional
info

RESULT:
A much more accurate estimate.
36th ADoDCAS

25

Resolving Concerns Over

a Data-Driven Approach

We dont know the size

Size by comparison against 10-100s of similar projects an accurate, proven
method.
A large number of sizing methods offers many solutions to project scoping.

Theres not enough past data

Numerous estimating methods (least absolute error, etc) are specialized for
small samples.

The data is very noisy

Models & methods are carefully combined to produce a more robust estimate.
Weighted least squares and other techniques help.

We dont have any data

But the model does: several thousand, many from NAVAIRs own contractors.

Nothing may be relevant

Estimating methods dont work that way they can find relevance even in
different sorts of data.
SEER parametric modeling is the backup for when nothing else works!

30 January, 2003

36th ADoDCAS

26

Midcourse Calibration

Calibration Dialog

Dialog lets
estimator input
earned value
information for
auto updating

Quick Estimate

MS
Project

Activity Duration Chart

Design

Phase duration
estimates are
updated on the
chart, with risk
ranges (in red)

Prototype
Spec
Code
Unit Test
Midcourse Calibration

______________________________________________________________________________________
Month
3
6
9
12
15
18
21
24

Potential variation in duration

30 January, 2003

36th ADoDCAS

27

wInsight Earned Value Will

Integrate With NexGen

30 January, 2003

36th ADoDCAS

28

Custom Lifecycles:
Data-Driven or User-Defined

Lifecycle Designer

Example - add FQT or

systems integration.

Activities and labor allocations can be

formulaic, fixed, or automatically
determined by ProjectMiner.

30 January, 2003

36th ADoDCAS

29

Accurate Scheduling Feature

A set of estimated
WBS elements

A set of properly
scheduled elements
Determine
Determinewhich
which
tasks
tasksshould
shouldfollow
follow
which,
which,how
howcritical
critical
the
therelations
relationsare
are
Automatic optimal
scheduling

30 January, 2003

36th ADoDCAS

30

Software Estimation & Project

Management
MS Project
S/W Estimation

MS Project w/ Software Estimation

30 January, 2003

36th ADoDCAS

31

Estimation and Transformation of

Microsoft Project
Arrow in the Quiver
for Process
Improvement

Saves time by automatically

laying out highly tailored plans
in Microsoft Project

Automatically connects with

estimation tools to obtain
sophisticated software project
resource & schedule estimates

Encourages best practices and

uniform estimating by
connecting project planners with
enterprise information

Perfectly complements Project

2002s enterprise and usability
focus

30 January, 2003

36th ADoDCAS

32

Data-Driven Estimates Against Past

Projects

..\..\..\business\01-016
NAVAIR SBIR N01-020 Software Estimating\Deliverables and Briefings\August 28th Briefing\scatterplot_run.avi
30 January, 2003
36th ADoDCAS

33

Data Collection Activities

NAVAIR is soliciting additional software data
Will benefit the entire community
To participate contact:
Rick Holcomb
AIR-4.2 Cost Department
(301) 342-2450
holcombra@navair.navy.mil

30 January, 2003

36th ADoDCAS

34