Techniques for Calibrating COCOMO

Estimating Models
Dan Ligett
Softstar Systems
Ligett@SoftstarSystems.com
www.SoftstarSystems.com
(603) 672-0987
Copyright (c) 2001 So

Calibration Overview
Local Calibration
Whats best fit?
Compare tools & methods

Copyright (c) 2001 So

Whats a Local Calibration?

Calibrated to your completed projects
Best fit to your environment
New Effort or Schedule Equation

Multiplier Only
Multiplier and Exponent Base

Not adding a new Cost Driver

Copyright (c) 2001 So

Why do a Local Calibration?

Fixes a multitude of sins:
Unusual standard for counting effort

Hours per Staff-Month

What effort is included

Nonstandard cost driver definition

Local definition of ACAP High, etc.

Idiosyncratic counting of SLOC

Physical lines

As long as youre consistently wrong, youre not wrong!

Copyright (c) 2001 So

Why do a Local Calibration?

Accuracy. A local calibration means you get an
equation that is the best fit to YOUR
environment.
You may want to do several local calibrations
for different types of projects, or different tool
sets.

Copyright (c) 2001 So

What data do you need?

Actual SLOC
EAF & SCED setting, Scale Factors
Actual Effort, Hours per Staff-Month
Actual Duration (to calibrate schedule equation)
You need:
5 data points to calibrate the multiplicative constant
10 data points to calibrate the constant & exponent

Copyright (c) 2001 So

Calibration Subtleties
To combine data points in a calibration, they must
be comparable (apples & apples)
Hours per Person-Month

Must calibrate with Normalized Effort

Normalized Effort = Actual Effort * ( HPM / 152 )
Example: Actual Effort = 100 PM, local HPM = 140 hours / PM
Normalized Effort = 100 * ( 140 / 152 ) = 92.1 PM

SCED Cost Driver

Must calibrate with Normalized Duration

Normalized Duration = Actual Duration / Schedule Acceleration
Example: Actual Duration of 75 months, SCED Very Low
Normalized Duration = 75 / 0.75 = 100 Months

Copyright (c) 2001 So

How do you define Best Fit?

So what's a best fit? The usual scheme is to use least
squares.
But it's easy to imagine different definitions of "best fit".
Minimize the sum of absolute difference
Minimize the sum of the square of the difference
Real space COCOMO 81
Log space COCOMO II
Minimize the sum of the cube or fourth power of the
difference
Best Pred(.10) or Best Pred(.15)
Best least squares such that no project is underestimated (!)
Copyright (c) 2001 So

Sample data adapted with permission from Table 4.17 of Software Estimation with COCOMO II

Copyright (c) 2001 So

Sample COCOMO 81 calibration results

(adapted from a Calico report)
Actual
Estimated Percent
Proj Effort
Effort
Error
Error**2
(lnA-lnE)**2
1
1,854.55
2,043.44
10.19
35,678
0.00941
2
258.51
280.59
8.54
487
0.00672
3
201.00
231.32
15.09
920
0.01974
4
58.87
60.30
2.43
2
0.00058
5
9,661.02
9,935.17
2.84
75,160
0.00078
6
7,021.28
8,193.71
16.70 1,374,593
0.02385
7
91.67
115.74
26.26
579
0.05436
8
689.66
895.85
29.90
42,513
0.06842
9
9,000.00
7,207.94
-19.91 3,211,493
0.04930
MRE
14.65
Calibrated Muliplier =
Exponent Base =

2.9757
0.91

4,741,426

0.23316

Pred Pred Pred Pred

(.10) (.15) (.20) (.25)
Yes Yes Yes
Yes Yes Yes Yes
Yes Yes
Yes Yes Yes Yes
Yes Yes Yes Yes
Yes Yes

33%

44%

Yes

Yes

77%

77%

COCOMO 81 Style Calibration

COCOMO II standard

Copyright (c) 2001 So

10

Sample COCOMO II calibration results

(adapted from a Calico report)
Actual
Estimated Percent
Proj Effort
Effort
Error
Error**2
(lnA-lnE)**2
1
1,854.55
1,869.93
0.83
237
0.00007
2
258.51
256.76
-0.68
3
0.00005
3
201.00
211.68
5.31
114
0.00268
4
58.87
55.18
-6.26
14
0.00418
5
9,661.02
9,091.58
-5.89
324,267
0.00369
6
7,021.28
7,497.98
6.79
227,244
0.00431
7
91.67
105.91
15.54
203
0.02086
8
689.66
819.78
18.87
16,931
0.02987
9
9,000.00
6,595.91
-26.71 5,779,655
0.09658
MRE
9.65
Calibrated Muliplier =
Exponent Base =

2.723
0.91

6,348,668

0.16230

Pred Pred Pred Pred

(.10) (.15) (.20) (.25)
Yes Yes Yes Yes
Yes Yes Yes Yes
Yes Yes Yes Yes
Yes Yes Yes Yes
Yes Yes Yes Yes
Yes Yes Yes Yes
Yes Yes
Yes Yes

66%

66%

88%

88%

COCOMO II Style Calibration

COCOMO II standard

Copyright (c) 2001 So

11

Sample calibration results, BEST MRE

Actual
Estimated Percent
Effort
Effort
Error
Error**2
(lnA-lnE)**2
1,854.55
1,854.11
-0.02
0
0.00000
258.51
254.59
-1.52
15
0.00023
201.00
209.89
4.42
79
0.00187
58.87
54.72
-7.06
17
0.00535
9,661.02
9,014.68
-6.69
417,753
0.00479
7,021.28
7,434.56
5.89
170,802
0.00327
91.67
105.02
14.56
178
0.01848
689.66
812.85
17.86
15,175
0.02701
9,000.00
6,540.12 -27.33 6,050,987
0.10193
MRE
9.48
Calibrated Muliplier =
Exponent Base =

2.700
0.91

6,655,008

Pred Pred Pred Pred

(.10) (.15) (.20) (.25)
Yes Yes Yes Yes
Yes Yes Yes Yes
Yes Yes Yes Yes
Yes Yes Yes Yes
Yes Yes Yes Yes
Yes Yes Yes Yes
Yes Yes Yes
Yes Yes

0.16295 67% 78% 89% 89%

Best MRE
COCOMO II standard

Copyright (c) 2001 So

12

Sample calibration results, Best Pred(.15)

Actual
Estimated Percent
Proj Effort
Effort
Error
Error**2
(lnA-lnE)**2
1
1,854.55
1,808.79
-2.47
2,094
0.00062
2
258.51
248.37
-3.92
103
0.00160
3
201.00
204.76
1.87
14
0.00034
4
58.87
53.38
-9.33
30
0.00959
5
9,661.02
8,794.32
-8.97
751,164
0.00883
6
7,021.28
7,252.83
3.30
53,615
0.00105
7
91.67
102.45
11.76
116
0.01236
8
689.66
792.98
14.98
10,674
0.01949
9
9,000.00
6,380.25
-29.11 6,863,065
0.11835
MRE
9.52
Calibrated Muliplier =
Exponent Base =

2.634
0.91

7,680,876

0.17224

Pred Pred Pred Pred

(.10) (.15) (.20) (.25)
Yes Yes Yes Yes
Yes Yes Yes Yes
Yes Yes Yes Yes
Yes Yes Yes Yes
Yes Yes Yes Yes
Yes Yes Yes Yes
Yes Yes Yes
Yes Yes Yes

67%

89%

89%

89%

Best Pred(.15)
COCOMO II standard

Copyright (c) 2001 So

13

How do you define Best Fit?

Figure of Merit
Best MRE
Best Error**2
(COCOMO 81)
Best (lnA - lnE)**2
(COCOMO II)
Best Pred(.15)

Pred Pred Pred Pred

Multiplier MRE Error**2 (lnA-lnE)**2 (.10) (.15) (.20) (.25)
2.7000 9.48 6,655,008
0.16295
67
78
89
89
2.9757 14.65 4,741,411

0.23316

33

44

78

78

2.7230 9.65 6,349,044

0.16230

67

67

89

89

2.6340 9.52 7,680,876

0.17224

67

89

89

89

Copyright (c) 2001 So

14

Comparing the schemes

COCOMO 81 calibration minimizes error in real space
COCOMO II calibration minimizes error in log space
Using a different definition of Best Fit resulted in a 10%
difference in the calibrated multiplier
MRE is better for COCOMO II style (in this example)
Pred() is better for COCOMO II style (in this example)
Some of these stats (e.g. MRE) require guess-and-check
Some are dangerous (e.g. PRED(.15) discards an outlier!)
Statistics lets you make an informed choice
Copyright (c) 2001 So

15

Calibration Tools
USC Tool
Calico
Spreadsheet

http://www.softstarsystems.com/calico.htm

Homegrown spreadsheet or program

Based on COCOMO II cookbook presented here

Copyright (c) 2001 So

16

Calibration with USC Tool

Free
Widely available
Defects

Doesnt handle HPM properly

Schedule Equation wrong

Copyright (c) 2001 So

17

Calibration with Calico

Free
Downloadable from www.SoftstarSystems.com
Handles HPM & SCED problems
Nice statistics and report
Pick & choose projects from your History files
Converts Costar estimates to history files

Computes EAF for you, which can be hard with components

Creates model files with new equations for Costar

You can use it as a standard data collection tool
Copyright (c) 2001 So

18

Calibration with Spreadsheet

Free
Downloadable from www.SoftstarSystems.com
You can dissect it
You can customize it
You can add your own statistics
Doesnt handle HPM or SCED nuances

Copyright (c) 2001 So

19

Calibration with a Homegrown Tool

Multiplier-only Calibration (COCOMO II style)

Details on next slide

See spreadsheets for an example:
www.softstarsystems.com/calico.htm

Multiplier-only Calibration (COCOMO 81 style)

Details in Software Engineering Economics

See spreadsheets for update to COCOMO II

Multiplier & Exponent Calibration

Details in Software Engineering Economics

See spreadsheets for update to COCOMO II

Copyright (c) 2001 So

20

COCOMO II style Multiplier Calibration

n

x ln( Actual _ Efforti )

i 1
n

i 1

j 1

y ln( EAFi ), EAFi EM i , j

SFi
z Exponent _ Base
* ln( KSLOCi )

100
i 1
A e ( x y z ) / n calibrated multiplier
n # projects, d # cost drivers
n

Copyright (c) 2001 So

21

Conclusions
Local calibration is good for you.
With most data sets, all of the definitions of best
fit will yield similar results. But, check the stats.
Start collecting data now.
As few as 5 data point will get you started.
Multiplier-only calibration is most conservative.
Use a good tool to CALIbrate COcomo.
Copyright (c) 2001 So

22