Practical Guide

to

Auditing the Software Development Process

November 22, 2008

Bruno Collet, MBA

Independent Project Manager & Auditor
bruno@brunocollet.com
www.brunocollet.com

Owner, Synapsys Canada

www.synapsys.ca

Check the free Agile audit tool Agile Benchmark at www.agilebenchmark.com

 TABLE OF CONTENTS
Introduction ___________________________________________________________ 3
Definition__________________________________________________________________ 3
Top three principles of auditing _______________________________________________ 3
The auditor ________________________________________________________________ 3
SD audit and transition management ___________________________________________ 4
Phases ____________________________________________________________________ 4
Initiating the audit ______________________________________________________ 4
Defining the mission _________________________________________________________ 4
Identifying the stakeholders __________________________________________________ 5
Planning the audit __________________________________________________________ 5
Kicking-off the audit ________________________________________________________ 5
Analyzing the organization _______________________________________________ 6
General information about the organization _____________________________________ 6
Role of SD in the organization_________________________________________________ 6
Technologies, platforms, tools, and paradigms ___________________________________ 6
Evaluating SD process ___________________________________________________ 7
Sources of information_______________________________________________________ 7
Client perspective ___________________________________________________________ 8
Team perspective ___________________________________________________________ 8
Management perspective ____________________________________________________ 10
Evaluating organizational structure _______________________________________ 10
Evaluating tools and technologies_________________________________________ 11
Testing evaluation results _______________________________________________ 12
Reporting audit results__________________________________________________ 12
Structure of the audit report and presentation __________________________________ 12
How to choose and assess alternatives _________________________________________ 13
Following up _________________________________________________________ 14
Plan of engagements________________________________________________________ 14
Champions _______________________________________________________________ 15
Coaching _________________________________________________________________ 15
Conclusion ___________________________________________________________ 15
References ___________________________________________________________ 16

Practical Guide to Auditing the Software Development Process

© Bruno Collet, MBA, Independent Management Consultant 2
Introduction

The purpose of this document is to summarize the process for auditing software
development (SD) practices. Although business and academic sources extensively
cover both auditing and SD processes, there is surprisingly little information about audit
applied to SD. This document briefly describes the SD audit process based on publicly
available information as well as personal experience in SD auditing and in IT project
management.

Definition
The purpose of an SD audit is to assess SD practices in the target organization in order
to recommend improvements to help the organization achieve its business objectives.
Although the scope of an SD audit can vary, it typically encompasses SD processes,
organizational structure, and tools and technologies. In essence, the goal of the SD audit
is to optimize SD's effectiveness and efficiency.
Note that an SD audit is not an IT audit. Although there is some overlap, the IT audit
deals primarily with IT operations.

Top three principles of auditing

We should keep in mind the top three principles of auditing:
1. Unaudited information lacks sufficient credibility to form a reliable base for
decision making.
2. Auditing requires more than common sense. It requires a systematic approach in
gathering evidence, and specialized knowledge (SD management in our case).
3. Auditors must be independent from the organization they audit. Their work must
be carried out objectively and impartially.

The auditor
The SD auditor should have the following characteristics:

− Excellent communication and interpersonal abilities

− Management consulting skills to formulate recommendation based on clear
evidence and flawless analysis techniques
− Experience in SD management, as project manager for example
− Experience in IT management, as development or IT manager for example

Practical Guide to Auditing the Software Development Process

© Bruno Collet, MBA, Independent Management Consultant 3
SD audit and transition management
Although it is sometimes independent, the SD audit is often closely related to transition,
change, and crisis management. The sequence is as follows:

SD audit and transition management

1. Crisis: the organization faces SD problems that threaten its business objectives.
2. Audit: the organization performs an SD audit to identify the causes of the
problems and to find solutions.
3. Transition and change: the organization implements the recommended solution.
The transition can range from non-disruptive changes (a series of minor
improvements), to specific changes (a small number of significant changes), to
full-fledged transition (changes that deeply affect SD processes, structures, and
possibly other business processes).

Phases
This document describes the main steps of the SD audit process, which are similar to
those of the traditional audit process:
1. Initiating the audit
2. Analyzing the organization
3. Evaluating SD processes
4. Evaluating organizational structure
5. Evaluating tools and technologies
6. Testing evaluation results
7. Reporting audit results
8. Following up

Initiating the audit

Defining the mission

The auditor and the sponsor have to define the scope of the audit. The SD audit will
always involve SD processes, but may or may not involve organizational structure, tools,

Practical Guide to Auditing the Software Development Process

© Bruno Collet, MBA, Independent Management Consultant 4
and technologies. The mission definition must also highlight the objectives in terms of
success factors.
Additionally, the auditor and the sponsor have to agree on the time and resources
needed to complete the audit.

Identifying the stakeholders

Stakeholders typically include:

− Management
− Auditors (internal and external)
− Employees, also referred as "the team", that consists in analysts, developers,
testers, project managers, and so on
− Sponsors

Planning the audit

The auditor has to plan the phases and activities of the audit. The audit can be seen as
a project in itself. As such, it has to be planned, and has milestones and objectives.
Milestones usually correspond to the release of status and intermediate reports to the
sponsor.

Kicking-off the audit

It is critical to communicate the audit goals and schedule to all stakeholders. Indeed,
poor communication will put some stakeholders on edge and will considerably hinder the
audit process. It is the auditor's and sponsor's responsibility to communicate properly to
ensure commitment from the start. Kicking-off the audit usually takes the form of a short
meeting with all stakeholders.
Beside mission specifics, the kick-off should clearly emphasize:

− Achieving the goals also means improving everyone's everyday job and creating
opportunities for further personal development.
− Assessment focuses on processes and structure. The audit does not judge
individuals' performance. Although this is not entirely true, it has to be
communicated this way.
− Assessment is 360 degrees (see further).

Practical Guide to Auditing the Software Development Process

© Bruno Collet, MBA, Independent Management Consultant 5
Analyzing the organization

It is a common pitfall to study SD in isolation. The auditor should always keep in mind
the role of SD in the organization and, more specifically, how SD contributes to the
organization's success.

General information about the organization

General organization's information includes:

− Vision and mission

− Market and competitive advantage
− Situation, including financials
− Structure

Role of SD in the organization

The most straightforward method for assessing the role of SD is to determine the value it
brings to other business processes.

− Core business processes: supply, production, marketing, sales

− Supporting business processes: general administration, human resources,
technical support
IT companies selling software solutions constitute a special case; for them SD is a core
strategic process at the heart of the production core business process.
We also have to stress that SD is not just about software development. It is about the
implementation of any information system, which can be made in part or in whole of
external software applications, components, or services. Software development does not
only refer to programming; it refers to the development of software solutions. Actually,
programming might (and often should) only be a small part of the organization's software
solutions.

Technologies, platforms, tools, and paradigms

The auditor has to determine the technologies used in SD. Technologies can be
categorized as base, key, pacing, and emerging. For example, an organization could be
relying on Java Enterprise as a base technology, and on XML-based Web Services as
an emerging technology in the software solutions.
Platforms refer to operating systems, servers, and channels, such as MS Windows, the
Web, and IBM Websphere server.
Tools refer to the basic tools used for SD in the organization, such as MS Project, IBM
Clearcase, and IBM Websphere Developer Studio.

Practical Guide to Auditing the Software Development Process

© Bruno Collet, MBA, Independent Management Consultant 6
The auditor should avoid getting bogged down in technical detail, though. On the other
hand, the auditor has to gain a good understanding of the paradigms that SD relies on in
the organization. Example paradigms are software-oriented architecture (SOA),
enterprise application integration (EAI), and software-as-a-service (SaaS). The
paradigms heavily influence SD processes.

Evaluating SD process

The evaluation is usually the phase that takes up the most time and effort. The
evaluation is based on a 360 degrees assessment that takes multiple perspectives into
account, as illustrated by the figure below.

360 degrees evaluation

Consolidating these four perspectives gives a reliable description of the reality.

Sources of information
At the heart of the audit are the different ways the auditor collects information. The
auditor mixes formal/informal, explicit/implicit, and verbal/written sources of information,
in order to create a reliable representation of the reality that will be the base for a sound
recommendation.
We summarize the sources as follows:

− Documentation: written documents or web pages provided by the organization.

− Surveys: questionnaires filled in by each team member independently, without
the auditor being present. The survey provides the auditor raw data on which he
can base the subsequent interviews.
− Interviews: face-to-face or phone interviews with individuals. The purpose of the
interviews with team members is to dig deeper into problems that have been
discovered through the survey to get a qualitative feedback.

Practical Guide to Auditing the Software Development Process

© Bruno Collet, MBA, Independent Management Consultant 7
Client perspective
The auditor has to assess client satisfaction, and put that in relationship with the SD
process. Ideally, he will be allowed to directly contact some clients. Measuring client
satisfaction is the best way to assess the success of SD projects. The auditor should
simply ask the clients whether projects met the required quality, were delivered on time
and on budget.
After having assessed client satisfaction, we have to evaluate the client perception of
how the team manages the client touch points:

− Initiating the project

o Business case and requirements
o Project charter, including scope, schedule, and budget
− Milestones and releases: schedule and validation
− Changes
− Communicating project status
− Closing the project
− Maintenance: solving defects

Team perspective
"The team" refers to all people directly involved in the production of software in the
organization. It usually includes analysts, architects, designers, developers, testers,
integrators, and project managers.
Assessing the SD process from the team's perspective relies on (1) general information
about SD and organizational practices, and (2) comparison of SD processes with the
best practices.
As mentioned earlier, information can be collected through documentation, surveys, and
interviews.

Sample general questions

− Is the team using a formal SD process, based either on an existing methodology

or on internal practices?
− What tools are used for project management?
− Are clients/users satisfied with the results?
− Are managers satisfied with the team's results?
− How are time and schedules managed? Are projects delivered on time?
− How is the budget managed? Are project delivered on budget?
− How is quality managed? Is the client satisfied with the quality?
− How is communication managed? Does each member understand what he/she
has to do? Is the management aware of projects' situations (status, etc.)? Is the
client kept up to date?
− How are risks managed? Is there a response plan?

Practical Guide to Auditing the Software Development Process

© Bruno Collet, MBA, Independent Management Consultant 8
 − How are changes managed? Is there a change management process?
− How is the team structured? How are roles distributed among team members?

Using best practices methodology as a reference

We use best practices because we need a reference to evaluate SD practices against.
First we have to choose what we defined as best practices. Usually we rely on an
existing methodology that has been proved successful for SD and that makes sense for
the type of organization and project we are dealing with.
For example, for a bank with 50 people in SD and 1000+ days projects, we might choose
IBM Rational Unified Process. On the other hand, for a small IT consulting company with
multiple small teams of 3-6 members working on 20-to-200 days projects, we might
choose an Agile methodology such as OpenUP.
Note that we do not want to compare SD practices with the chosen best practice
methodology in details: the methodology is simply used as a starting point to define best
practices, in terms of roles, processes, activities, and deliverables.
Below we will describe sample evaluation guidelines based on IBM Rational Unified
Process (IBM RUP) and OpenUP. Refer to the documentation of these methodologies
for more details.

IBM RUP evaluation guidelines

When using IBM RUP as best practices, we can for example assess how SD practices
compare to IBM RUP's workflows. During the interview, the auditor can examine how the
team handles activities, artifacts, and roles associated with these workflows.
The IBM RUP workflows are:

− Engineering workflows:
o Business model
o Requirements
o Analysis and design
o Implementation
o Test
o Deployment
− Supporting workflows:
o Project management
o Configuration and change management
o Environment

OpenUP evaluation guidelines

When using OpenUP as best practices, we can for example assess how SD practices
compare to OpenUP practices. During the interview, the auditor can examine how the
team handles work products, tasks, guidance, and roles associated with these practices.
Alternatively, the auditor can base his assessment on the more traditional disciplines of
OpenUP.

Practical Guide to Auditing the Software Development Process

© Bruno Collet, MBA, Independent Management Consultant 9
The OpenUP practices are:

− Management practices:
o Iterative development
o Risk-value lifecycle
o Two-level project planning
o Whole team
o Team change management
− Technical practices:
o Concurrent testing
o Continuous integration
o Evolutionary architecture
o Evolutionary design
o Shared vision
o Test driven development
o Use case driven development

Management perspective
The auditor has to establish management responsibility in choosing SD processes, tools
and technologies, and in structuring the team.
Additionally, in the same fashion we did it for other perspectives, we have to look at
activities in which managers are involved, such as:

− Project initiating (effort estimation, resource allocation)

− Project closing
− Project reporting
− Billing
We can also collect information about client satisfaction, team structure, and
time/budget/quality, in the same way we did for the team perspective.

Evaluating organizational structure

Organizational structure is a topic in itself. Therefore this section is an extreme

simplification of how to assess and affect organizational structure for SD.
In many occurrences the auditor will notice that the organizational structure is not
optimal for the organization's SD activity. It often involves one or several of the following:

− Team does not cover skills, or some skills have no backup

− Power structure significantly differs from official organizational structure

Practical Guide to Auditing the Software Development Process

© Bruno Collet, MBA, Independent Management Consultant 10
 − Some skills are redundant
− Interpersonal problems
− Behavior problems
− Conflicts of interest
These issues tend to affect employee satisfaction, which leads to motivation and
productivity problems.
Solutions might require adapting the structure, such as:

− Transferring
− Changing roles
− Promoting
− Hiring
− Laying off
− Training
The question of outsourcing also often surfaces along an SD audit mission. Outsourcing
is a topic that we will not delve into.

Evaluating tools and technologies

Evaluating tools and technologies strongly depends on the specific context. Therefore,
we can only mention a few guidelines.

− How are tools and technologies chosen?

− Are the current technologies and tools efficient?
− Are current tools used and leveraged appropriately?
In particular, we want to avoid tools and technologies being chosen based on the fashion
of the day (how "cool" they are), based on some team members interest to develop
personal skills, or based on some managers relationships with suppliers. Other red flags
include dependence on a large number of technologies and tools, rogue tools, and
compatibility/interoperability issues.
In most cases, the auditor does not possess the skills required to assess technologies
and tools. He has to delegate investigation to internal auditors (who cannot be
stakeholders) or external specialists. However, a simple checklist will allow examining if
and how the organization addressed basic tools requirements:

− Project management tools

− Bug tracking tools
− Customer service or CRM tools
− Automated testing tools

Practical Guide to Auditing the Software Development Process

© Bruno Collet, MBA, Independent Management Consultant 11
 − Source repository or version control tools

Testing evaluation results

In the evaluation phase, the auditor lets others express themselves. Consequently, most
information collected so far comes from documents and people's perception. Although
the 360 degrees evaluation minimizes the risks for serious misperception, the evaluation
is not finished before the auditor can directly assess some elements. Direct assessment
consists in collecting facts first-hand, without intermediaries such as people or
documents.
In essence, testing evaluation results consists in gathering evidence to support the
evaluation, with focus on:

− Gaps: elements for which no significant evidence has been found.

− Inconsistencies: elements for which contradictory evidence has been found.

Reporting audit results

Reporting audit results consists in:

1. Submitting an audit report to management (a document), and presenting the
recommendation to management (a PowerPoint presentation for example)
2. Helping the management in making decisions based on the recommendation.
The management might also want to adjust the auditor's recommendation.
3. Presenting audit results, recommendation, and implementation plan to all
stakeholders (a PowerPoint presentation for example).

Structure of the audit report and presentation

The audit report and the presentation to management should have the following
structure:

− Audit goals and link with business objectives (1 slide)

− Audit process (1 slide)
− Analysis of the organization:
o Organization and market summary (1 slide)
o Role of SD in the organization (1 slide)
− Evaluation results:
o Client perspective (1 slide)

Practical Guide to Auditing the Software Development Process

© Bruno Collet, MBA, Independent Management Consultant 12
 o Team perspective (1-2 slides)
o Management perspective (1 slide)
o Consolidated results (1 slide)
− Solutions ("alternatives"): See next section.
− Recommendation (1 slide): summarizes why the recommended alternative is the
best one.
− Plan of action (1-2 slides): brief plan of action, with high-level timeline and tasks.
− Risk assessment (1 slide): indicate probability of occurrence, impact, and
contingency plan for each risk that can hinder the implementation of the solution.
The presentation should be around 20 slides long and last no more than 30 minutes.

How to choose and assess alternatives

Selection criteria
The auditor chooses 5-to-8 criteria the alternatives will be measured against. Some of
these criteria tend to appear in all audits, while some are project-specific.
Among the criteria that appear in all audits and recommendations, we find:

− Return on investment (short term and long term)

− Fit with resources
− Fit with skills and culture
− Fit with strategic vision
− Risks

Financials
No recommendation is complete without strong financial evidence. The auditor has to
estimate the costs and benefits of each alternative in dollar value. There are many
methods for analyzing ROI. One such method is to calculate the total value of
opportunity (TVO). Compared to traditional ROI-based methods, TVO has the advantage
to take opportunity costs into account and to rely on net present value (NPV) of cash
flows instead of accrual-based value.
Although cost and revenue items can vary according to the audit, some items are
remarkably invariable:

− Cost savings due to process improvement resulting in increased efficiency

− Additional revenue due to improved credibility and confidence
− Cost and forfeited revenue due to learning during implementation of solution
− Cost of auditing, training, and other consulting fees

Choice of alternatives
The auditor presents 3 to 5 alternative solutions. One should be status quo and the
others should all be valid solutions that significantly vary when measured to selection
criteria. It is important to mention the status quo alternative because we want to measure

Practical Guide to Auditing the Software Development Process

© Bruno Collet, MBA, Independent Management Consultant 13
alternatives against what we already know, which is the current situation. Note that all
alternatives have to be realistic in terms of resources.
For each alternative, the auditor has to indicate pros and cons.

Recommended solution
The auditor summarizes the evaluation of the alternatives against selection criteria. The
summary usually takes the form of a decision table with criteria as lines and alternative
as columns. Each cell quantifies how well the alternative measures against the criteria.
Measures have to be simple, such as a 0-to-5 rating for example. Criteria can also be
weighted individually.

1. Status Quo 2. Do This 3. Do That

ROI – 6 months ☺  
ROI – 3 years  ☺ 
Fit with skills/culture ☺  
Fit with strategy  ☺ 
Low risks  ☺ 
Recommendation
Example of simple decision table

Don't be fooled by the simplicity of this decision table. The goal of the audit is precisely
to summarize a complex situation in a form that facilitates decision making. Each of
these simplistic scores is the result of a thorough analysis. In other words, the auditor
must be ready to explain each of these scores in detail.
The auditor should not compromise on the results of his analysis. He should have
definitive arguments to support his recommendation and should never endorse a
solution that he doesn't believe in.

Following up

There is little chance that the audit recommendation will be carried on if the auditor does
not follow up on the audit. Three things can support the organization in implementing the
recommendation and reap the full benefits of the SD audit.

Plan of engagements
The auditor has to agree on a plan of engagements with the sponsor and management
of the organization. The plan of engagements determines subsequent reviews, for
example quarterly reviews during the next year. It focuses on controlling the risks that
were identified during the audit. In essence, an SD audit is an iterative project: it is

Practical Guide to Auditing the Software Development Process

© Bruno Collet, MBA, Independent Management Consultant 14
unlikely that heavy changes can be implemented quickly and completely right after the
audit. Therefore, it is imperative to plan these changes along a longer period of time.

Champions
The auditor has to raise champions to carry on his work in the organization. Indeed,
neither the auditor's distant follow-up nor management's commitment is sufficient to
keep up everyday's interest in the changes that have to be implemented after the auditor
leaves. Champions are employees in the organization who are convinced that
implementing the recommendation is a solution to the problem, and who have power to
make the recommendation a reality. This power often comes from influence. Therefore,
champions are ideally people who work in or with the team every day. Champions are
the auditor's best allies to overcome resistance to change.

Coaching
Good auditors will not be afraid of coaching the implementation of their recommendation.
They will switch from auditor to coach and work with the stakeholders to implement the
plan of actions. Coaching tends to diminish in intensity as the changes are implemented.
Alternatively, the auditor can recommend another consultant to do the coaching. Either
way, coaching significantly improves the speed and chances of success.

Conclusion

SD audit is not a trivial matter. It requires blending several disciplines in a cohesive

whole to establish a foundation for change. It borrows from software development,
project management, organizational behavior, management, and even managerial
finance.
Every organization facing potential or actual SD-related problems should seriously
consider an SD audit. The return on investment of an SD audit is not easy to evaluate,
but guesstimate is usually enough to demonstrate excellent return. Imagine improving
efficiency by 10% on all IT projects, and improving client satisfaction. Even small
organizations will notice that such relatively humble results will yield several thousands
of dollars in savings, as well as increased revenues. ROI is almost certainly positive and
payback period is usually in the near future.
However, an SD audit can fail. The main causes for failure are (1) a mission scope that
does not allow addressing the real problems, and (2) the inability to follow up on the
audit to implement the solution. For example, SD problems caused by dysfunctional
team structure will not be solved if processes are improved while the team structure
remains unchanged.
Without doubt, many organizations would benefit from SD audits. The impact of SD on
the bottom line is often underestimated. Moreover, many managers still believe that

Practical Guide to Auditing the Software Development Process

© Bruno Collet, MBA, Independent Management Consultant 15
general audits based on accounting practices cover the whole business, and fail to
understand that these audits do not address SD issues as they should.

References

Information technology audit, Wikipedia

http://en.wikipedia.org/wiki/Information_technology_audit
Software development process, Wikipedia
http://en.wikipedia.org/wiki/Software_development_life_cycle
Audit, Wikipedia
http://en.wikipedia.org/wiki/Audit
Business process, Wikipedia
http://en.wikipedia.org/wiki/Business_process
OpenUP, Eclipse
http://epf.eclipse.org/wikis/openup/
IBM Rational Unified Process, Wikipedia
http://en.wikipedia.org/wiki/IBM_Rational_Unified_Process
Improve Your Audit Interviews, ASQ Audit Quality Division, J. P. Russell (2006),
Regulatory Compliance Newsletter
http://www.gmplabeling.com/news_letters/spring_news_05_06.pdf
Interviewing as an Auditing Tool, Linda M. Leinicke, Joyce A. Ostrosky, W. Max
Rexroad, James R. Baker, and Sarah Beckman, (2005), CPA Journal
http://www.nysscpa.org/cpajournal/2005/205/essentials/p34.htm
How to Write a Survey or Questionnaire, eHow
http://www.ehow.com/how_16596_write-survey-questionnaire.html
Coaching, Wikipedia
http://en.wikipedia.org/wiki/Coaching
Putting a Price Tag on IT Projects, Bruno Collet (2008), CIO.com
http://advice.cio.com/bruno_collet/putting_a_price_tag_on_it_projects
The Total Value of Opportunity (TVO) of an IT Project, Bruno Collet (2008), CIO.com
http://advice.cio.com/bruno_collet/the_total_value_of_opportunity_tvo_of_an_it_project
Net present value, Wikipedia
http://en.wikipedia.org/wiki/Net_present_value
Related blog posts written by Bruno Collet:

Practical Guide to Auditing the Software Development Process

© Bruno Collet, MBA, Independent Management Consultant 16
 Effectiveness and efficiency,
http://www.brunocollet.com/blog/index.php?blog=5&title=effectiveness_and_effici
ency&more=1&c=1&tb=1&pb=1
How to avoid making unrealistic recommendations,
http://www.brunocollet.com/blog/index.php?blog=5&title=how_to_avoid_making_
unrealistic_recommen&more=1&c=1&tb=1&pb=1
The mission behind the mission,
http://www.brunocollet.com/blog/index.php?blog=5&title=the_mission_behind_th
e_mission&more=1&c=1&tb=1&pb=1
Guidelines to define a consulting mandate:
http://www.brunocollet.com/blog/index.php?blog=5&title=guidelines_to_define_a
_consulting_mandat&more=1&c=1&tb=1&pb=1

Practical Guide to Auditing the Software Development Process

© Bruno Collet, MBA, Independent Management Consultant 17