INCREASING YOUR

SOFTWARE DEVELOPMENT ROI

How to measure productivity — objectively
WHY SOFTWARE PRODUCTIVITY IS SO IMPORTANT
- AND SO HARD TO MEASURE

In some ways, a software devel- manage your business better by

opment project is like any other
initiative: You need to see strong
results for the effort and mon-
ey you put in, whether the code
comes from an outside vendor or
an internal development team.
The problem? Measuring the
output of software development
isn’t so simple.
Although software development
is often thought of as an engi-
neering discipline, the compar-
ison to traditional engineering
goes only so far. You’re not pro-
ducing widgets on an assembly
line, or building a road. A house
cannot be generated. A bridge
cannot be copied.
answering vital questions like
these:
• Do our vendors deliver
high-quality software fast
enough?
• Are we spending too much on
developing this application?
• Are our technologies hinder-
ing our development speed?
• Do our development teams
become more efficient over
time?
Ultimately, it all comes back to a
single guiding question:
Do I get enough value for the
money I spend on development?

To evaluate software projects ac-

curately, you need deep visibility
into both inputs and outputs—
something you can’t get with
traditional approaches. Once
you have that clearer view, you’ll
WHY TRADITIONAL PRODUCTIVITY MEASURES DON’T STACK UP
The (Supposed) Alchemy of Software
Development
Software developers often dislike be-
ing measured, and in many cases be-
lieve that it’s impossible to quantify
true coding skill. Indeed, they have a
point: A lot of traditional measures
do fall short of that goal. But even if
we grant that development teams
need a certain amount of freedom
in executing their tasks, coding isn’t
some magical process.
Typically, companies relying on com-
plex development projects don’t take
into account how the software is con-
structed, the actual effort expended
by developers, or the long-term qual-
ity of the software produced.
But all of these can be measured—if
you know how.
The Problem with Functional Testing
Most software development con-
tracts focus on functional results that
can be defined for the purposes of
acceptance testing. Function points
have their place (“Yes, the module
performs the defined function”), but
they don’t show you enough.
Vital aspects such as performance,
security, maintainability, and relia-
bility are not functional. Buying soft-
ware without measuring for them is
like paying a custom-made car with-
out ever opening the hood.
And you will definitely pay for it:
Low-quality software is very expen-
sive to operate, maintain, and up-
date, and that’s before you think of
the opportunity costs of operating
less efficiently than your competi-
tors.
WHAT YOU SHOULD BE MEASURING INSTEAD
- INPUT
Traditional measures mostly ig-
nore the amount of actual de-
velopment effort that goes into
producing code. From the per-
spective of a vendor or coder,
that’s fine: If you can produce
good code more efficiently, you
should.
But if you’re the one paying for a
project, don’t you want to know
if the work is being achieved
with half the effort?
Focusing only on the end prod-
uct allows software suppliers
to price their services based on
the perceived value they deliver,
instead of on the cost of devel-
opment. Insight into the actual
effort involved to deliver a piece
software allows for a more bal-
anced discussion between ven-
dor and client.
It’s worth noting that many or-
ganizations don’t do a very good
job of gathering and tracking
consistent data on inputs. Thus,
drawing attention to the issue at
all may improve data collection.
And while it may always be hard
to determine productive inputs
for a given day or week, it should
become easier over time to un-
derstand how much work can
realistically be done during, say,
a calendar quarter.
WHAT YOU SHOULD BE MEASURING INSTEAD
- OUTPUT

As already noted, functional testing isn’t adequate

for measuring development output. Raw measures
such as lines of code written are also unreliable in
isolation, because a few high-quality, high-relevance
lines are worth more than hundreds of low-quality,
low-relevance ones. That’s why you need to measure
output on three separate dimensions:

1. Production — Analysis of code added, changed,

and deleted allows production to be expressed in
programmer-months of work. This must be con-
sidered in tandem with quality to account for the
code’s maintainability.

2. Quality — Any code of inferior quality contains

technical debt, which we’re able to calculate in
terms of programmer-months of postponed fu-
ture work. Subtracting that debt from production
yields net production—a far more useful meas-
ure.

3. Relevance — The best code in the world, even

delivered ahead of schedule, is meaningless if it
doesn’t meet your business needs. Basic metrics
like story points should be combined with code-
based automated measurements as well as ad-
vanced measures framed explicitly in financial
terms, such as business value per change.
HELPING CLIENTS DRIVE A HARD BARGAIN
When you have clear visibility for
the relevant inputs and outputs
of software quality, it’s much
easier to establish common
ground and keep development
teams accountable.
Our approach is specifical-
ly geared to enable three key
things:
1. Challenging vendor quotes
— Project doomed to go
past its timetable and over
its budget.
2. Calculating quality of work
delivered — Whether you’re
dealing with an entire pro-
ject, an individual developer,
or even a single code com-
mit, knowing for certain how
good a body of code is gives
you incredible leverage so
you can manage better.
3. Calculating amount of work
performed — This gets back
to what was said previously
about inputs from your ven-
dors. Don’t you want to know
if you’re paying for 100 units
of work that requires only 10
units of effort?
Recently, SIG helped a client
uncover a troubling decline in
productivity from a vendor. At
first, the vendor had delivered
code at a consistent level of
production and quality; when
the client suspected that
something had gone amiss,
they asked us to investigate.
Within two weeks, SIG ex-
perts verified a clear dropoff in
productivity tied to a change
in the vendor’s allotment of
offshore personnel. SIG also
uncovered problems in how
the vendor was documenting
its work. Our holistic view of
software development pro-
ductivity—backed by the ex-
pertise and tools needed to
implement it—allowed the
client to recover €30,000 in
fees from the vendor.
HELPING DEVELOPERS & VENDORS DEMONSTRATE THEIR VALUE
Good vendors and internal development teams
should also see the value of a better approach
to measuring software productivity and qual-
ity. Under traditional methods, a vendor might
hide how little effort or quality code went into
a project to inflate the fee; conversely, though,
the client might drastically underestimate the
difficulties that the vendor had to overcome to
deliver good code.

Using the better approach advocated here, both

vendors and clients can work with greater trans-
parency and trust. When real value is delivered,
it’s going to be more clearly understood than
ever. And when problems are uncovered, they
will be much easier to diagnose and fix.

People running software development teams,

whether internal or external, have a vested in-
terest in achieving greater efficiency, rewarding
outstanding programmers, meeting real busi-
ness needs, and building a strong reputation for
delivering top value for money.
 
WHAT YOU SHOULD BE MEASURING INSTEAD
- OUTPUT

If you’re concerned with the productivity of your

software development—and you should be—
you need the deep code visibility that gives you
a penetrating view into how your software is
made, how well it works, and how robust it will
be over time.

Developing that visibility and using consistently

will help you measure the quality of each ven-
dor’s work, trends in productivity, the impact of
internal improvement programs, and more.

Beyond that, it will clear away the fog from an

operational area that can either be a source of
strategic advantage . . . or the cause of many
woes.

Functionality

Implemen-
tation Proposed approach

Traditional observer Software system

WHAT’S NEXT
There’s no reason to start panicking. We happen
to know a brilliant team who can help you get
it right. In the meantime, you can start thinking
about getting your whole IT landscape right. We
have some inspirational reads that may get you
going.

SIG gives technology leaders the visibility they

need to address current software problems and
prevent future ones from ever happening.
Drawing on proprietary methods and decades of
expertise, SIG helps organizations fundamen-
tally improve the security and performance of
the enterprise applications that support every
aspect of their businesses. Headquartered in
Amsterdam, SIG serves companies in Europe and
beyond. Information about SIG and its work can
be found at www.sig.eu