A Tale of Two Companies

A Tale of Two Companies
A Narrative Story Based

Explanation of “Why” Organizations are Struggling with
Transformations such as Agile and DevOps.
Thomas Gilmore
Creative Open Source, LLC
257 Dunham
Monte Vista, CO 81144
Copyright © Thomas Gilmore
All rights reserved. No part of this book may be reproduced, stored in a retrieval
system, or transmitted in any form or by any means, without the prior written
permission of Creative Open Source, except in the case of brief quotations embedded in
critical articles or reviews.
Every effort has been made in the preparation of this book to ensure the accuracy of
the information presented. However, the information contained in this book is sold
without warranty, either express or implied. The author shall not be held liable for any
damages caused or alleged to be caused directly or indirectly by this book.
The author has endeavored to provide trademark information about all of the
companies and products mentioned in this book by the appropriate use of capitals
All rights reserved, for permission to Reproduce selections from this book, write to
Permissions, Creative Open Source, LLC 257 Dunham, Monte Vista, CO 81144
First Edition
Printed in the United States of America
3
Credits
Author
Thomas Gilmore
Reviewers
Eric Buckeye
Mandana Nabizadeh
Ryan Fritzsche
Josue Aparicio
Stephen Rufle
Nancy Grande
Alfredo Herrera
Proof Readers
Mandana Nabizadeh
Greg Tanner
4
Author Biography
Tom is an Agile/DevOps Transformation Coach. Tom brings a unique

background to coaching, having started his career working in manufacturing
operations management where he worked closely with Lean Manufacturing
and TQM concepts. Tom’s educational studies have helped foster these Lean
foundational beliefs, with a BA in Operations Management and an MBA in
Technology and Innovation Management.
With over 20 years of experience in Lean Manufacturing, TQM, Agile

Development, DevOps, Quality Assurance, Agile Testing and Automated
Testing, he has helped coach many organizations with their move towards
Agile and DevOps philosophies in order to become Learning Organizations.
Tom has worked closely with individuals, teams and management to coach
them on the intricacies of Organizational Behavior and Development in order
for the organization to create a social environment where learning can take
place. Tom’s coaching focuses on helping the organization create a cultural
climate that allows individuals to improve, because as individuals get better
at what they do, teams naturally improve. And when teams improve,
organizational results dramatically improve.
5
Dedication
Firstly, I would like to say a big thank you to my amazing wife, Tanya who
has had to put up with many days of her husband glued to his computer
screen while he wrote this book. I am so grateful for you holding down the
fort while I finished the book.
I would also like to thank my son Nathan who has tried to understand why
daddy sometimes couldn’t play with him, you are the light of my life and I
hope to make up for the time I have spent working on the book and not
playing with you.
6
Introduction
This book is an attempt to help explain why so many organizations have

failed in their attempts at Agile and DevOps, and why so many will in the
future. This book helps to bridge the knowledge gap between organizations
who wish to learn how to successfully move to Agile and DevOps and the
organizations who have successfully made the move to Agile and DevOps.
The book introduces learning points in simplified form, so that concepts are
clearly spelled out in order for both technical and non-technical individuals
to understand.
Audience
This book is intended for anyone, technical or non-technical who would

like to learn how to help move their organization forward in their Agile or
DevOps journey.
Book Structure
This book follows the story of two organizations competing in the same
industry who have slowly found themselves struggling to survive. As we
follow the two organizations on their journey to survive, we have section
“breakdowns” where the reader can jot down notes about their observations
from that section. There is no right or wrong in terms of what you have
observed in each section, the section may have highlighted issues you have
seen inside your organization or during your career. Finally, at the end of
each section we have Learning Points where we discuss subjects in a
simplified manner.
7
Table of Contents
A Tale of Two Companies ............................................................................................................................................................................................................................................. 13
Let’s Summarize what we know and have learned: .................................................................. 15
Learning Point(s): .................................................................................................................................................................................................................................................. 15

Why So Many Fail at Agile and DevOps ................................................................................................................................................................................................ 15
Understanding Agile and DevOps.............................................................................................................................................................................................................. 17
Start with Active Inertia ................................................................................................................................................................................................................................... 18
Organizational Transition, It’s the Downfall for Most.................................................................................................................................................................. 20
Dr. W. Edwards Deming .................................................................................................................................................................................................................................. 23
The Journeys Begin.......................................................................................................................................................................................................................................................... 33

Orca Explores their Problem’s .................................................................................................................................................................................................................................. 34
Breakdown of Orca’s Problems: ............................................................................................................. 35
Learning Point(s): .................................................................................................................................................................................................................................................. 35

System of Profound Knowledge ................................................................................................................................................................................................................... 36
Eliyahu M. Goldratt – Theory of Constraints (5 focusing steps) .......................................................................................................................................... 37
Lean Manufacturing as a guide .................................................................................................................................................................................................................. 40
Bernoulli's Principle ............................................................................................................................................................................................................................................ 43
Orca Evaluates Their Choices ................................................................................................................................................................................................................................... 44
Breakdown of Orca’s Choices: ................................................................................................................. 45
Learning Point(s): .................................................................................................................................................................................................................................................. 46

Conflict Creates Failure.................................................................................................................................................................................................................................... 46
Technical Debt ........................................................................................................................................................................................................................................................ 48
Orca Makes a choice........................................................................................................................................................................................................................................................ 49
Breakdown of Orca’s Choice: ................................................................................................................... 51
Learning Point(s): .................................................................................................................................................................................................................................................. 52

Deming’s Deadly Diseases of Management......................................................................................................................................................................................... 52
Choosing an Agile Methodology .................................................................................................................................................................................................................. 54
Dealing with the fear of change .................................................................................................................................................................................................................. 55
The Implementation begins ........................................................................................................................................................................................................................................ 58
Breakdown of Orca’s Implementation: .............................................................................................. 60
Changing the Culture ........................................................................................................................................................................................................................................ 61

Management by Objectives ............................................................................................................................................................................................................................ 64
Agile Upstream and Downstream .............................................................................................................................................................................................................. 66
Orca Must Make Tough Decisions ......................................................................................................................................................................................................................... 68
Breakdown of Orca’s Dilemma: .............................................................................................................. 71
Learning Point(s): .................................................................................................................................................................................................................................................. 71

Why we Unit Test .................................................................................................................................................................................................................................................. 72
Breaking down silos ............................................................................................................................................................................................................................................ 74
The need for feedback loops .......................................................................................................................................................................................................................... 76
Orca and Dolphin’s Project ......................................................................................................................................................................................................................................... 77

Let’s Examine Orca’s Strategy in Depth ............................................................................................................................................................................................................ 78
Iteration 1................................................................................................................................................................................................................................................................................ 78
Breakdown of Iteration 1: ........................................................................................................................... 78
Learning Point(s): .................................................................................................................................................................................................................................................. 79

DevOps Infrastructure ...................................................................................................................................................................................................................................... 79
DevOps SCM Strategy ....................................................................................................................................................................................................................................... 81
Iteration 2................................................................................................................................................................................................................................................................................ 83
8
Learning Point(s): .................................................................................................................................................................................................................................................. 84

Tightly Coupled Code Issues......................................................................................................................................................................................................................... 85
Overcoming the Cultural Roadblocks ..................................................................................................................................................................................................... 87
Flipping the Testing Pyramid........................................................................................................................................................................................................................ 89
Components of Change..................................................................................................................................................................................................................................... 91
Iteration 3................................................................................................................................................................................................................................................................................ 94
Learning Point(s): .................................................................................................................................................................................................................................................. 95

Using Kaizen to Fight Waste ......................................................................................................................................................................................................................... 96
Moving from a Ticket Taking Culture ..................................................................................................................................................................................................... 99
Agile Best Practices .......................................................................................................................................................................................................................................... 100
Iteration 4............................................................................................................................................................................................................................................................................. 103
Breakdown of Iteration 4: ........................................................................................................................ 103
Learning Point(s): ............................................................................................................................................................................................................................................... 104

Specification by Example ............................................................................................................................................................................................................................. 104
Define CI and CD ............................................................................................................................................................................................................................................... 106
Shift Left and Continuous Testing ...................................................................................................................................................................................................................... 107

..................................................................................................................................................................................................................................................................................................... 107
Let’s Take a Break and Play a Game ................................................................................................................................................................................................................. 109
The Envelope Game ..................................................................................................................................... 109
Iteration 5............................................................................................................................................................................................................................................................................. 111
Learning Point(s): ............................................................................................................................................................................................................................................... 112

Hidden Constraints .......................................................................................................................................................................................................................................... 112
Iteration 6............................................................................................................................................................................................................................................................................. 114
Learning Point: .................................................................................................................................................................................................................................................... 115

Conway’s Law ....................................................................................................................................................................................................................................................... 115
Iteration 7............................................................................................................................................................................................................................................................................. 116
Learning Point(s): ............................................................................................................................................................................................................................................... 117

Versioning ............................................................................................................................................................................................................................................................... 118
Change Management....................................................................................................................................................................................................................................... 120
Provisioning ........................................................................................................................................................................................................................................................... 121
Iteration 8............................................................................................................................................................................................................................................................................. 123
Learning Point(s): ............................................................................................................................................................................................................................................... 124

Blame Game .......................................................................................................................................................................................................................................................... 124
Creating Culture Change through Organizational Behavior............................................................................................................................................................. 127
What we will learn ........................................................................................................................................ 128
The Mirror Image Fallacy ............................................................................................................................................................................................................................. 128

Threats to Rational Decision Making ................................................................................................................................................................................................... 129
Decision Making Illusions: .......................................................................................................................................................................................................................... 129
Understanding the Social Environment ............................................................................................................................................................................................. 130
DevOps Targets the Symptoms of Stress .......................................................................................................................................................................................... 131
9
Barriers to Effective Communication ................................................................................................................................................................................................... 131
DevOps Communication Roles ................................................................................................................................................................................................................. 133
Dealing with Fear............................................................................................................................................................................................................................................... 133
How to Deal with Negative Politics: ....................................................................................................................................................................................................... 134
Resolving the Blame Game with Superordinate Goals ............................................................................................................................................................ 135
Want to do DevOps? Better Understand Organizational Development ........................................................................................................................ 135
The Five Features of Organizational Development: .................................................................................................................................................................... 136
Sources of Resistance to the Change Agent .................................................................................................................................................................................... 137
Three Categories of Change: ...................................................................................................................................................................................................................... 137
Sources of Change Agent Resistance:.................................................................................................................................................................................................. 138
Fighting Resistance with Forcefield Analysis ................................................................................................................................................................................. 140
Overcoming Forces of Resistance Identified in Forcefield Analysis: ............................................................................................................................... 141
Organizational Culture Versus the Informal Organizational Culture............................................................................................................................ 142
The Elements of Organizational Culture ........................................................................................................................................................................................... 143
Cross Cultural Communication............................................................................................................................................................................................................... 143
Let’s take a look at Dolphin now .......................................................................................................................................................................................................................... 144

Dolphin Explores their Problem’s ........................................................................................................................................................................................................................ 144
Breakdown of Dolphin’s Problems: ................................................................................................... 145
Learning Point(s): ............................................................................................................................................................................................................................................... 146

Denialism ................................................................................................................................................................................................................................................................ 146
Listening .................................................................................................................................................................................................................................................................. 148
ChatOps ................................................................................................................................................................................................................................................................... 151
Dolphin Looks for Answers ...................................................................................................................................................................................................................................... 152
Breakdown of Dolphin’s Answers: ..................................................................................................... 153
Learning Point(s): ............................................................................................................................................................................................................................................... 154

SWOT Analysis .................................................................................................................................................................................................................................................... 154
Value Proposition Cards ............................................................................................................................................................................................................................... 156
Using Survey Feedback in DevOps to Stimulate Information Sharing ......................................................................................................................... 158
Dolphin Reviews the Evidence ............................................................................................................................................................................................................................... 159
Breakdown of Evidence Review: .......................................................................................................... 161
Learning Point(s): ............................................................................................................................................................................................................................................... 162

DevOps Business Advantages ................................................................................................................................................................................................................... 163
Become a Story Teller ..................................................................................................................................................................................................................................... 165
Dolphin Learns about Agile and DevOps ....................................................................................................................................................................................................... 167
Breakdown of what Dolphin Learned: ............................................................................................ 169
Learning Point(s): ............................................................................................................................................................................................................................................... 169

DevOps as an Extension of Agile ............................................................................................................................................................................................................ 169
It’s about Culture!!! .......................................................................................................................................................................................................................................... 171
Dolphins Ah Ha Moment ........................................................................................................................................................................................................................................... 173
Breakdown of the Ah Ha Moment: .................................................................................................... 174
Learning Point(s): ............................................................................................................................................................................................................................................... 175

Organizational Structures ........................................................................................................................................................................................................................... 175
IT as Core Business Component ............................................................................................................................................................................................................. 179
IT is the new Factory Floor.......................................................................................................................................................................................................................... 180
Dolphin puts Their Strategy in Place ................................................................................................................................................................................................................ 183
Breakdown of the Strategy: .................................................................................................................... 185
Learning Point(s): ............................................................................................................................................................................................................................................... 186

Value Stream Mapping................................................................................................................................................................................................................................... 186
Murphy’s Curve................................................................................................................................................................................................................................................... 188
10
Single versus Double Loop Learning .................................................................................................................................................................................................... 190
Theory of Constraints and Five Focusing Steps ........................................................................................................................................................................... 191
Evangelize, Evangelize and keep doing it .......................................................................................................................................................................................... 193
The Difficult Choices Dolphin Initiates ............................................................................................................................................................................................................ 197
Breakdown of Dolphin’s Choices ........................................................................................................ 198
Learning Point(s): ............................................................................................................................................................................................................................................... 199

Everyone is Responsible for Quality ..................................................................................................................................................................................................... 199
Are you Siloed?.................................................................................................................................................................................................................................................... 201
Tackling Training ............................................................................................................................................................................................................................................... 202
Orca and Dolphin’s Project ...................................................................................................................................................................................................................................... 203

Let’s Examine Dolphin’s Strategy in Depth .................................................................................................................................................................................................. 204
Iteration 1............................................................................................................................................................................................................................................................................. 204
Learning Point(s): ............................................................................................................................................................................................................................................... 205

Telemetry and Monitoring ............................................................................................................................................................................................................................ 206
Why TDD ................................................................................................................................................................................................................................................................. 208
Rev Up Quality with Code Reviews ........................................................................................................................................................................................................ 210
Maximize Post Mortems................................................................................................................................................................................................................................. 212
Iteration 2............................................................................................................................................................................................................................................................................. 213
Learning Point(s): ............................................................................................................................................................................................................................................... 215

True Automated Testing................................................................................................................................................................................................................................ 215
Pipeline Swarming............................................................................................................................................................................................................................................. 217
It’s all about Feedback Loops .................................................................................................................................................................................................................... 218
Iteration 3 ............................................................................................................................................................................................................................................................... 219
Learning Point(s): ............................................................................................................................................................................................................................................... 221

DevOps Orchestration Versus Automation...................................................................................................................................................................................... 221
Quality Gates ........................................................................................................................................................................................................................................................ 222
A/B Testing ............................................................................................................................................................................................................................................................ 223
Managements Role in Agile and DevOps ........................................................................................................................................................................................... 225
Iteration 4............................................................................................................................................................................................................................................................................. 228
Learning Point(s): ............................................................................................................................................................................................................................................... 230

Striving for Green .............................................................................................................................................................................................................................................. 230
DevOps = Continuous Learning Organization ............................................................................................................................................................................... 231
Created a New DevOps Department? Go Back! ............................................................................................................................................................................ 236
Iteration 5............................................................................................................................................................................................................................................................................. 239
Learning Point(s): ............................................................................................................................................................................................................................................... 240

Andon Culture ..................................................................................................................................................................................................................................................... 240
DevOps Metrics ................................................................................................................................................................................................................................................... 242
Artifact Repository............................................................................................................................................................................................................................................. 245
Iteration 6............................................................................................................................................................................................................................................................................. 246
Learning Point(s): ............................................................................................................................................................................................................................................... 247

Dojos and Centers of Excellence ............................................................................................................................................................................................................. 247
Security and Compliance ............................................................................................................................................................................................................................. 248
11
Evolutionary Development/Architecture ........................................................................................................................................................................................... 250
Microservices ........................................................................................................................................................................................................................................................ 252
..................................................................................................................................................................................................................................................................................................... 252
Containerization ................................................................................................................................................................................................................................................. 253

Cloud .......................................................................................................................................................................................................................................................................... 255
Iteration 7............................................................................................................................................................................................................................................................................. 256
Learning Point(s): ............................................................................................................................................................................................................................................... 257

DevOps Release Strategies .......................................................................................................................................................................................................................... 257
Simian Army ......................................................................................................................................................................................................................................................... 260
False Prophets ..................................................................................................................................................................................................................................................... 262
Let’s Summarize Orca’s and Dolphin’s Probable Outcomes: ............................................................................................................................................................ 263
Conclusion.............................................................................................................................................................................................................................................................. 264
12
A Tale of Two Companies
Our story begins with the plight of two companies, Orca and Dolphin and their struggle
to not only remain relevant in their market, but to simply survive. These two fictional
businesses started out just like many companies do, as an idea by their founders who
had a vision combined with ambition and a willingness to work hard.
After many years and countless hours of hard work and dedication by a core group of
employees, the dreams of both companies were realized. Both Orca and Dolphin are
recognized as leaders in the travel industry. For a number of years both Orca and Dolphin
successfully grew their businesses and continued to gain market share. With the success
of both companies came the need to not only grow their locations, but also the number of
employees needed to meet their customer’s needs. The management of both companies
have now become consumed with growing profits, in order to do this, they increasingly
implemented new rules, bureaucracy, processes and focused on cutting costs.
Many of the core employees who were responsible for growing both companies now
lament the changes they are seeing to the culture of their companies. In the early days of
both companies, employees felt a sense of ownership and freedom to work as a team to
overcome issues and continuously innovate. During the early days of the companies,
individuals, groups and teams were given the freedom to experiment with new processes
and tools in order to not only overcome obstacles, but to improve the services and products
they provided without any sense of fear. If it was determined that a new technology could
improve the current technology framework, employees were given the time, resources or
training needed to implement the new technology. With management’s new short-sighted
focus on just growing profits by cutting costs and adding layers of bureaucracy the culture
of innovation has been replaced by a culture of command and control driven by fear.
The teams at both companies suddenly start to find it increasingly more and more
difficult and painful to deliver quality software products and services to their customers.
The management at both companies initially put together task forces to implement new
processes with the goal to help improve quality, deliver software more quickly and reduce
risk. The reality, as they came to find out, is that it does the opposite, it makes the task
of delivering software even more laborious and more complex. At the same time, the
management at both companies are now threatened with a new imposing threat, several
new startups in their market space have suddenly appeared and to their shock are not
only gaining market share but are stealing market share from them at a rate that puts
their very existence at risk.
13
At this point, both companies have come to an evolutionary crossroads that will require
them to make decisions that will determine if they will continue to survive or as the famous
quote from Dr. W. Edwards Deming states “Learning is not compulsory, but neither is
survival”. As both companies are faced with these new risks, they must come to a critical
decision, do they continue to live with the way they do things and attempt to tweak them,
or do they try and get back to the days of their origins where they were lean, nimble and
able to innovate.
A few challenges stand in the way of both companies being able to decide which path
they should follow in order to not only regain the competitiveness that they initially had
in their early days, but to also survive. The top management at both companies is made
up of engineers who, while initially having great ideas for a new company, lack the
business acumen to compete in this new era and managers brought in with business
backgrounds, have little to no knowledge of the IT industry. The management at both
companies must make a critical decision to either attempt to turn the ship around
themselves, possibly hire individuals who might possess the skills and knowledge to help
turn the ship around or bring in consultants to help guide them. Various managers at
both companies over the last few years have heard of techniques and methods to improve
quality and enhance processes, such as, Lean, Agile, Continuous Delivery, Continuous
Integration, Six Sigma, TQM, PDCA, ITIL and ITSM. Over the last few years some of the
top management has even attempted to implement these techniques and methods in the
companies by tasking lower level management and leads to drive these efforts, in one of
these two companies you can still see the posters for PDCA (Plan, Do Check, Act) that
were put up during the attempt to implement PDCA. These efforts to implement these new
ideas have failed due to a number of factors, such as lack of support by top management,
a lack of knowledge by those implementing these ideas and those who have dictated these
new efforts. In addition, the cultures that have grown over the last few years have tended
to kill any new attempts at change. As these new ideas have consistently failed, they have
also had the direct effect of causing resistance to grow among the employees who were
mandated to use them.
14
Let’s Summarize what we know and have learned:
1.
2.
3.
4.
Learning Point(s):
Why So Many Fail at Agile and DevOps
There are numerous reasons that most companies fail in their Agile and DevOps
transformations, here we will take a look at some of the most common causes. If we do
some research on transformations what we discover is that historically almost 70% of all
change projects (yes, Agile and DevOps are change projects) across companies of all types
and sizes have failed in the past. The number one reason for this failure is the company’s
15
failure to recognize that with all change projects comes a corresponding organizational
transition. We will discuss organizational transition in depth in an upcoming learning
point, but what we need to know is that the organizational transition involves changing
relationships, new skills needed, and cultural changes brought about by the Agile and
DevOps change projects. If companies fail to first take into account the organizational
transition, then they are most likely to fail at Agile and DevOps. This is shown in research
by Gartner “90 Percent of I&O Organizations Attempting to Use DevOps Without
Specifically Addressing their Cultural Foundations Will Fail” and for Agile in the research
by Steve Denning “For all the talk about Agile, only a small percentage—less than 10%—
of Agile implementations are fully Agile in the true sense of the word”1.
Another issue we need to remember is that all new revolutions – methodologies -

practices generate rhetoric. The Vendors attempt to create awareness of their products.
Consultants, Academics, and Leaders/Journalist all need to feed their respective
distribution channels with information or new ideas. When we take a look at the Agile and
DevOps communities what we see is lots of exuberance being thrown about by the Agile
and DevOps, but what we don’t see is a focus on the foundations that Agile and DevOps
are built on and also very little attention to the sciences and concepts that have a
tremendous impact on Agile and DevOps, such as Active Inertia, Organizational Behavior,
Organizational Development and Organizational Learning among others.
If we take a look around the Agile and DevOps landscape, we will see that a couple of
vendors have offered up frameworks intended to help companies with their Agile and
DevOps transformation. What some have learned the hard way when implementing these
frameworks is that they have lots of holes in them and its becomes a small miracle if they
can actually bring about the full Agile and DevOps transformation these companies were
looking for. Among the problems with these frameworks are that they completely fail to
even take into account the organizational transition piece, or call out specifically how
organizational learning takes place, or how the teachings of someone like Dr. Deming need
to be applied and even how to get the company to change their mindset in terms of how
IT is viewed inside the company.
As we move through the book, we hope to introduce the reader to many of these ideas,
concepts and tools that have unfortunately been neglected by the Agile and DevOps
communities.
1 Denning, Steve - For Agile, It's The Best And The Worst Of Times, Forbes 2018
16
Understanding Agile and DevOps
One of the common mistakes individuals, teams and organizations make when it comes
to DevOps is looking for some kind of definitive example of what exactly Agile and DevOps
are. It is not uncommon to hear at a conference, attendees lamenting that they have not
yet heard a presentation that defines or clarifies the exact steps needed to implement Agile
and DevOps. All too often we look for some kind of paint by numbers or step by step guide
that will magically guide the organization towards Agile and DevOps. The reality is that
Agile and DevOps should be viewed as more of a philosophy and not some strictly defined
method that once completed signifies we have attained Agile and DevOps. By viewing them
as more of a philosophy, it allows organizations to better understand that their journey
will be different than other organizations due to their cultural issues, communications
barriers, products and architecture.
By looking at Agile and DevOps as more of a philosophy, we can then attempt to

implement it in terms of being an organizational vision and goals. That means Agile and
DevOps are not looked at as an IT initiative, but instead are treated as an organizational
strategic vision and goals. In the Learning Points that follow, we will make the case for
Agile and DevOps being treated as an organizational vision and goals and highlight that
IT must not be looked at as a support role in the organization, but instead as a core
component of the organization.
17
Start with Active Inertia
Donald N. Sull, associate professor of management practice at London Business School, is author
of“Why good companies go bad and how great managers remake them” (Harvard Business School
Press, 2005)
Most of us have either seen the various State of Agile or DevOps Reports, research on
Agile and DevOps, or we have seen firsthand the extreme difficulties that companies are
experiencing with their attempts at Agile and DevOps. Many of these companies struggling
are well established successful companies, which begs the question, why are so many
struggling with Agile and DevOps? The answer may lay in the concept of Active Inertia, a
concept created by Professor Donald Sull from the MIT Sloan School of Management.1 Dr.
Sull is recognized as one of the leading global experts on strategy execution.
18
So, what is Active Inertia and how could it be playing a role in so many established
companies struggling with Agile and DevOps? Active Inertia occurs when companies
attempt to respond to disruptive shifts like Agile or DevOps by accelerating activities that
succeeded in the past. When these companies are presented with changes like Agile or
DevOps, these organizations trapped in Active Inertia simply do more of the same, in other
words they run their traditional playbook. They may do it a little quicker perhaps or tweak
it, but basically, they try and implement Agile or DevOps just like they have with any
previous change.
What causes these companies to get stuck in a rut and not be able to respond with the
changes necessary to effectively adopt Agile or DevOps? The answer is actually surprising,
the very same commitments that allowed the company to be successful in the first place
now become shackles that hamper the change necessary for Agile or DevOps. Many of us
have heard the old adage that success often breeds complacency, in reality there is a very
real link between these company’s early success and their subsequent struggles with
change. The commitments that were required for their early success harden over time and
eventually work to constrain the company’s ability to change when environmental shifts
such as Agile or DevOps come on the scene.
These initial commitments made by successful companies were required as part of their
initial success, but they tend to harden over time. The initial success these companies
enjoyed leads to an atmosphere that reinforces management’s belief that all they need to
do is reinforce this successful formula. Over time and through the ingraining of the
formula into the culture, the company stops considering alternatives to their
commitments, and simply starts taking them for granted. The components of the success
formula over time grow less flexible, and eventually leads to what termed the “Dynamics
of Failure”.
History is littered with companies who have been affected by Active Inertia, many are
well known such as IBM, Microsoft, Compaq, Xerox and Firestone. The reality we see is
one in which companies instead of making the cultural, structural or process changes
necessary to gain the benefits offered by Agile or DevOps, instead attempt to force Agile or
DevOps to fit into their traditional formula of doing things. Until companies awake to the
realities of Active Inertia, the success rates for Agile and DevOps will continue to be
abysmal.
19
1Donald N. Sull, associate professor of management practice at London Business School, is author of
“Why good companies go bad and how great managers remake them” (Harvard Business School Press,
2005)
Organizational Transition, It’s the Downfall for Most
Unfortunately, the DevOps movement is marked by the vast majority of organizations

failing in their attempts at DevOps. Gartner points out that 90% of organizations who fail
to first deal with their cultural issues fail in their DevOps attempts. The way this is worded
is a little misleading, because it gives the sense that there is some small portion of
organizations who are failing to deal with culture and the majority have successfully dealt
with their cultural issues. It’s actually the opposite, the vast majority of organizations fit
into the category of failing to deal with their cultural issues and will most likely fall into
the 90% failure category.
So, the question is, why is it so difficult for organizations to deal with their cultural
issues? The reason for this is actually due to the lack of education inside organizations
and many of the tool vendors and consultants leading the DevOps movement. You can go
to a number of web sites and read about how the vendors/consultants can help you with
your DevOps Change or Transition. But guess what, Organizational Change and
Organizational Transition are not the same thing. This lack of understanding is the main
culprit in organizations not making the necessary cultural changes in order to allow
themselves any chance at adopting DevOps.
You are probably asking yourself what the heck is the difference between Change and
Transition that could possibly cause all these failures. Let’s explore Organizational
20
Change first. Change simply happens when something starts or stops, or when something
that previously used to happen in one way starts happening in another way. It can happen
at a particular time, or in several stages at different times. Organizational Change is
structural, economic, technological or demographic, and it can be planned or managed
generally using some type of rational model. Simply put, we are talking about change such
as starting automated testing, implementing a CI tool set, doing Value Stream Mapping,
implementing Scrum or hiring more resources.
Organizational Transition on the other hand is broken into a three part psychological
process that extends over a long term period and cannot be planned or managed by the
same rational models used with Change. We will briefly cover the three phases at a high
level, covering the three phases in detail can’t be covered in this learning point.
Phase One – “Individuals have to let go of the old situation and of the old identity that
went with it”. No one can begin a new role or have a new purpose if that person has not
let go of the old role or purpose first. A great deal of what we call resistance to change is
really difficulty with the first phase. So, what are we exactly talking about here? Some
examples would be the shifting role or lack of role that Quality Assurance finds themselves
in with the move to automated testing or developers now being asked to start handling
automated testing as part of their role and the integration of dev and ops.
Phase Two – Individuals have to go through what’s termed the “neutral zone” between the
old reality and new reality that is most likely still not very clear. In the neutral zone
individuals tend to feel a loss or sense of confusion, where new ideas alternate with a
sense of meaningless and a time when many simply go through the motions. But, this is
also the time when the real transition is taking place. One way in which organizations can
help individual’s in the neutral zone is with the use of Dojo’s. But remember, a Dojo will
only be successful if you create the needed Social Environment with all its elements.
Phase Three – Individuals have to make a new beginning that is much more than the
relatively new start required in a change. This new beginning may involve learning new
competencies or skills, establishing new relationships (After the breakdown of silos),
becoming comfortable and familiar with new processes and procedures and learning to
think in new ways (Andon culture). It is this three phase reorientation from old to a new
way that DevOps leaders, management and the organization overlook when they plan a
change such as DevOps (this applies to Agile as well).
21
Borrowing from a line from Harry Levinson the industrial psychologist, executives have
been wise about the mechanics of change and stupid about the dynamics of transition.
That stupidity is dooming many of their change efforts to failure.
22
Dr. W. Edwards Deming
Deming, W E. Out of the Crisis. Cambridge, Mass: Massachusetts Institute of Technology, Center
for Advanced Engineering Study, 1986. Print.
23
It is highly recommended that any organization facing challenges such as Orca and
Dolphin, fall back on the management guidelines offered by Dr. W. Edwards Deming in
his works. The work of Dr. Deming has served as a guideline for countless organizations
over the years and has earned Deming the title of “The Father of the Quality Revolution”.
By following Dr. Deming’s famous 14 Points, that were originally presented in “Out of
the Crisis”, companies will find that this helps cultivate a fertile culture in which a more
efficient workplace, higher profits, and increased productivity may grow.
Applying Deming’s 14 Points to Agile and DevOps:
Create Constancy of Purpose toward improvement of products and services, with the aim
to become competitive and to stay in business, and to provide jobs.
It can be argued that the drive to remain competitive and stay in business is driving
the adoption of Agile and DevOps? Many organizations today look at the need to adopt
them for several reasons, first someone in their vertical may have been successful in
adopting Agile and DevOps and now has a distinct competitive advantage. A second reason
is that a company may fear that someone outside of their vertical may have adopted them
and now is looking to move into new verticals due to the increased efficiencies and
increased capacity they have gained through Agile and DevOps (Amazon’s acquisition of
Whole Foods in an example of this). Lastly, many companies are finding that their
customers have either adopted Agile and DevOps or are the path to adopting them and
are now expecting their vendors to also be moving towards Agile and DevOps in order to
provide them with the quality software in a timely manner they expect.
Planning for quality in the long term is not only exposed by Deming but is really a
fundamental of what we are trying to accomplish with Agile and DevOps. One of the key
points missed by many individuals when discussing why things like automated testing,
shifting testing left, telemetry and controlling technical debt are needed for Agile and
DevOps is the lack of understanding that if the software products being developed don’t
have a long enough life span, then the company will not fully experience the ROI on these
products/services that they should be.
A common issue in many companies today is that they see a problem with a process
and attempt to fix the process over and over, but, instead of trying to fix the process they
should first investigate if this is the correct process in the first place. This is a case of
24
attempting to move the culture in the company from single loop learning to double loop
learning which we will discuss later in the book.
Unfortunately, in today’s business environment most management is unable to resist

reacting with short term solutions for processes that actually require management to
think in the long term. A classic example of this is automated testing. All too often as an
automated testing effort is being implemented, the management will delay/stop the
automated testing efforts at the first fire that pops up, thus killing all momentum that the
individuals may have built up.
Adopt the new Agile and DevOps philosophy. The traditional Software Development model
has changed, and management must awaken to this new challenge and learn new
responsibilities and help lead the change.
The new philosophy of Agile and DevOps require that companies move to replace
traditional command and control management and instead take on a role of clearing any
obstacles that teams may face in their efforts to take ownership of products/services
(management can almost be viewed as playing a Scrum Master role). This is in line with
DevOps attempting to move Software Development from a ticket taking culture support
role to being an integral component of the organization's core competencies.
Beginning with Agile and continuing with DevOps, these philosophies have preached a
cooperative leadership model to help break down the silos that have been built up over
the years in most companies. The reason we need a cooperative management model is so
that we can replace all the individual processes that these silos have built and taken
ownership of over the years and replace them with a single process that is owned by
everyone.
Agile and DevOps should not be viewed as just changing Software Development but
changing the entire way organizations operate. DevOps is having a profound effect on
areas such as Marketing, Sales and Finance. Why you ask is DevOps affecting all of these
areas, successful DevOps implementations bring about the ability not only release
software in a fraction of the time traditional Software development has been able to, but
even the release strategies have changed. Some companies find that they no longer follow
a traditional major release strategy every couple of years, but instead are not only
continually releasing new software, but even running experiments to gauge if their
customers like the new changes or features. This has a direct effect on how to market the
25
software, how do you sell when there are possibly no major releases and how you do
financial projections.
Cease dependence on inspection to achieve quality. Eliminate the need for inspection on
a mass basis by building quality into the product in the first place.
At the heart of Lean, Agile and DevOps is the belief that you don’t test after the fact,
but instead should build quality into the process from start to finish through the idea of
“shifting testing left” and “Continuous Testing”. At the forefront of this is the struggle with
automated testing and the role quality assurance plays in the Agile and DevOps models.
The facts are quite simple, we know that manual testing as performed by traditional
quality assurance is expensive, unreliable and most importantly does not improve quality,
it simply points out the lack of quality. The need for automated testing of not only the
applications we are building is needed, but automated testing of the infrastructure used
to build the software is needed. This automated testing must not only be built into the
process from the beginning but needs to continue into production. There are some truths
here that unfortunately are being ignored by most organizations, first, that automated
testing requires programming experience and the majority of it must be done at the Unit
level which only developers can do. What this means is that all organizations must face
the issue of what role does traditional quality assurance play if they do not have the
required skills necessary to perform this automated testing.
End the practice of awarding business on the basis of price tag. Instead, minimize total
cost.
All too often management today is so focused on the bottom line ROI that they forget
about the top line ROI. This is often seen in the rush to offshore software development or
testing work and not realizing that while it may seem to be cost effective, in an Agile and
DevOps world this may result in unseen costs that directly affect the top line ROI. Agile
and DevOps are built around the philosophy of releasing small incremental releases to
customers quickly and through the use of telemetry, monitoring and A/B testing quickly
making any changes identified by the customer to fully meet their needs.
This becomes increasingly difficult if we develop the software internally and then have
to send it to an offshore team who must perform either manual testing or must write the
automated tests after the fact. If an offshore team writes the software, we may find a
26
situation where the internal testing is in a holding pattern waiting for the offshore
development team to deliver the software for it to be tested.
Improve constantly and forever the system of production and service, to improve quality
and productivity, and thus constantly decrease costs.
A key component of Deming’s work is the concept of PDCA (Plan, Do, Check, Act), this
concept is also a fundamental part of DevOps. In DevOps, we create what are termed
Feedback Loops, which are in fact, teams practicing PDCA.
A critical philosophy that has come out of TPS (Toyota Production System) is the idea
of Andon Cord. The Andon Cord philosophy has been effectively adopted at companies
like Amazon, where it has had incredible results. At Amazon, when a customer service
agent gets a phone call from a customer explaining that there is a problem with the
product he or she has just received. If it’s found to be a repetitive defect, they empower
the customer-service agent to “stop the line,” which means taking the product off the
website until the defect is fixed. The objective is to start the line again, once the defect has
been resolved.
Adopting the Andon Cord philosophy has had an amazing impact at Amazon, it has led
to the elimination of tens of thousands of defects per year. Another key benefit is that
Andon Cord has empowered frontline workers (Helps move the transformation to an
ownership culture). The authority to stop the line is an enormous proof of trust for
customer-service agents, who usually have no real authority to help upset customers.
Another way in which DevOps is borrowing from Deming’s 14 points is the idea of not
only moving towards automated testing but shifting all of this testing left in order to gain
immediate feedback early in the software development process. Deming’s belief that you
should build quality into the product/service and not test it into the product/service,
helps move the software industry back into alignment with true quality assurance.
Unfortunately, over the years the idea of quality assurance has been hijacked by many in
the industry to become quality control.
27
Institute training on the job.
One of the exciting things to come out of the DevOps movement is the realization by
more and more organizations (just as we witnessed during the Lean Manufacturing
revolution during the 1980’s) that large scale training is not only required for DevOps, but
ultimately helps to move a company towards becoming a learning organization. What
companies are learning is that moving towards DevOps is not easy and that unless you
retain your top talent and give them time to not only learn these new tools and processes
your doomed to failure on your DevOps journey. Luckily a number of new innovative
learning models are coming out of DevOps, such as Dojo’s at Target and Centers of
Excellence.
Many organizations over the last few years have learned an expensive lesson, by trying
to bring in outside resources to solve issues like automated testing. The lesson these
companies have learned is that first, there are very few resources in the market place that
actually know how to do automated testing correctly and lastly if they did find a resource
who was knowledgeable, the resource would eventually leave and take all the knowledge
with them leaving the company back at square one minus a large chunk of money.
Institute leadership. The aim of supervision should be to help people do a better job.
Supervision of management is in need of an overhaul, as well as supervision of staff
workers.
This is an issue that aligns with breaking down barriers between departments and
working on self-organizing teams in Agile. Traditional management in an Agile and
DevOps environment need to understand that when individuals under their direct
supervision are working on Agile teams, it should be and is the Agile team who is really
responsible for confirming that individuals are pulling their own weight and conforming
to working effectively with the team. Traditional department supervisors should take on
more of a Scrum master or coach role to help individuals reach their potential by helping
them gain any training needed or helping them resolve any team dynamic issues that
might arise. In other words, these traditional supervisors need to move away from the idea
that they need to inspect the individual work of their direct reports, have these individuals
provide detailed reporting of the work/progress or put their department above the team.
28
Drive out fear, so that everyone may work effectively for the company
The Agile and DevOps movement has developed about a number of ideas for driving out
fear in organizations in order for the Agile and DevOps movement to flourish. One idea
that has been instituted by a number of companies is the Agile idea of Blameless Post
Mortems. The idea behind this is to signal to employees that the ultimate goal is to achieve
high quality by doing things correctly and that these Blameless Post Mortems are the
perfect place to understand how mistakes happened and instead of placing blame, learn
from them. The DevOps movement has helped usher in the idea that it’s ok for teams or
individuals to fail, as long as they fail fast and then learn from it.
One of the realities that many in the industry are starting to learn on their Agile and
DevOps journey is that there is no one way to implement Agile and DevOps, one size fits
all set of tools or blueprint for creating a delivery pipeline, what these individuals and
companies are learning is that Agile and DevOps are about experimenting, learning from
the experiments and using what was learned to plan the next experiment without the
cloud of fear hanging over teams and individuals.
Break down barriers between departments.
Many individuals in the industry believed that the Agile movement would be the driver
to break down the traditional barriers (silos) between departments. Originally, many of
the early Agile practitioners thought that by moving Programmer’s, Quality Assurance and
Business Analysts out of the traditional departments and putting them together on Agile
teams, we would finally break apart these siloed departments and this would lead to teams
who owned products/services and would eventually break the ownership by silos of tools
and processes. An example of this belief by many was that programmers and quality
assurance would all eventually morph into developers. The reality is none of this has
happened in the industry as a whole.
You will hear much talk in the DevOps movement about DevOps being an extension of
Agile, or simply that DevOps is Agile being applied to the entire organization. The truth is
probably more in line with DevOps being a reaction to the failure of the Agile movement
over the years. The hard truth is that Agile, while not formally calling for many of the
DevOps building blocks such as continuous integration, automated testing and the
breaking down of silos, the early practitioner believed they would come naturally through
29
the move to the Agile process much like occurred in Lean Manufacturing. It should be
noted that the failure of the Agile movement was not a result of the original practitioners
or the fundamentals of Agile, but the lack of will by the industry to make the cultural
changes necessary to be successful. One way to look at this is Agile was telling the
industry in a calm voice they needed to make cultural changes, while DevOps is giving
the same message in a much louder voice.
Eliminate slogans, exhortations, and targets for the work force asking for zero defects and
new levels of productivity. Such exhortations only create adversarial relationships, as the
bulk of the causes of low quality and low productivity belong to the system and thus lie
beyond the power of the work force
The DevOps movement changes how we look at metrics and which metrics we value.
One way of understanding this is that Agile and DevOps do not value looking at how many
issues were uncovered in the development pipeline, instead we are really worrying about
how fast we are able resolve any issues detected.
In Agile and DevOps, we are focused on preventing any issues as early in the process
as possible since this is significantly less expensive than catching an issue later in the
pipeline. If we can prevent issues early in the process, then this generally signals that we
are building a process that is able to self-detect issues, instead of needing to rely on
manual quality control phases or worse feedback from the customer.
The responsibility of supervisors must be changed from sheer numbers to quality.
The DevOps movement requires that a mindset shift must occur in all organizations,
instead of solely focusing on the sheer amount of software being pushed through the
development pipeline, these companies must shift to focusing on the quality being
delivered to their customers. The reason for this is that the clear majority of organizations
do not realize that a large portion of this work being pushed through the development
pipeline is waste! Most industry research has found that companies are spending almost
70 to 80 percent of their software development budgets on maintaining existing code.
What this means is that large portions of the work being pushed through the development
pipelines at these companies is simply maintenance work, instead if these companies were
concentrating on quality, they would be pushing new revenue generating work to their
customers.
30
Remove barriers that rob people in management and in engineering of their right to pride
of workmanship. This means abolishment of the annual or merit rating and of
management by objective.
There is simply no way of getting around the fact that to allow the Agile and DevOps
movement to grow and flourish, you must change the way you incentivize employees.
Incentivizing must change to encourage employees to not only experiment with new tools
and processes, but more importantly that they take ownership and see these new tools
and processes through to their conclusion whether that be success or failure.
Many in the DevOps movement will discuss that almost all successful DevOps
movements in companies have started from within and weren’t driven from the top by
management. While this is to a true point, the reality is that if top management had not
put the conditions in place to allow DevOps to prosper these companies would probably
never have been successful. What are these conditions, having an incentive structure in
place that encourages experimentation, provides the resources necessary for teams and
individuals to be able to learn about new tools and processes and providing an
environment that individuals involved in the DevOps movement can do so without fear.
Institute a vigorous program of education and self-improvement.
A common theme being driven into the industry by many placement firms is that the
way to overcome the DevOps hurdle is to go out and hire several DevOps engineers or full
stack engineers who can come in and almost overnight jump start you on the way to
DevOps. The sad truth is that almost across the board these placement firms have no idea
what DevOps actually, is and that there simply aren’t enough DevOps/full stack engineers
who know what they are doing available in the market. The truth is that you can’t hire
your way to DevOps, instead you need to improve the current skills of your workers.
You accomplish this by encouraging your people to learn new skills in order to prepare
for future changes and challenges, by doing this you help employees build skills to make
the employees more adaptable to change, and better able to find and achieve
improvements.
31
Put everybody in the organization to work to accomplish the transformation. The
transformation is everybody's job.
One of the 14 points from Deming that unfortunately has not been understood by a
large number of organizations moving towards DevOps is that DevOps should involve
everyone. Too many organizations today are making the unfortunate mistake of thinking
the move to DevOps requires them to create a new DevOps department and to assign
DevOps titles to the individuals working in this new department. This is tremendous
mistake that in almost all cases is leading to the failure of DevOps in these organizations.
As was discussed in an above point about breaking down barriers between

departments, the last thing we would ever want to do while moving towards DevOps is to
add a new department and the barriers that go along with it. Many organizations who
have been successful in moving to DevOps have even gone as far as adding Marketing,
Sales and HR individuals to the teams working on implementing DevOps in order to help
spread the concepts of DevOps and values behind it. One idea behind this is that in order
to successfully scale DevOps you need to evangelize the progress and concepts of DevOps,
so in order to do that it often makes sense to include team members from other areas of
the organization who know how to do this and can offer these contributions to the team
in order to help seed their success.
32
The Journeys Begin
At this point the journeys for both companies diverge, with each company attempting
to take a different path to survival. We will follow each company to see how they went
about determining their path. The difficulties they experienced along the way, and finally
we will follow both companies as they undertake a project to hopefully prove that they
have been able to successfully turn their companies around.
33
Orca Explores their Problem’s
The CEO of Orca while reviewing the bleak numbers being reported by finance and sales
comes to the determination that he needs to act swiftly to turn the fortunes of Orca
around. The CEO setups an offsite emergency retreat for the executive management team
for the following week and sends out an email detailing why they are holding the retreat
and for everyone to come prepared to share their solutions to the problems Orca is facing.
The following week, the entire executive management team meets at their offsite
location to attempt to solve the issues facing Orca. The CEO starts the meeting by
explaining that he would like each manager to present their solutions they have prepared
and that after all the solutions have been presented, the team will explore each solution
one by one and come up with their plan of attack.
As the day progresses, the management team presents various solutions to turn the
company around. One of the first solutions presented is the idea of dramatically cutting
costs by offshoring large parts of the development and quality assurance work. As part of
this solution, it is also thought that this in turn may also make Orca nimbler since the
lower costs of resources would allow Orca to have more development and testing resources
when needed, thus allowing for elastic capacity.
Another possible solution that is pitched is for Orca to invest in a more modern
marketing campaign to better compete against these new upstart competitors who Orca
management believes are partially gaining market share because of their edgy marketing
campaigns. The Marketing team also shares their suggestions for taking advantage of
marketing on sites such as Facebook, LinkedIn and other popular web sites.
Lastly, the CTO of the Orca makes a pitch for making the company, in particular, the
IT department Leaner by adopting the Agile methodology. The CTO explains that he
believes that unless the IT department adopts these Lean principles of software
development they are doomed to extinction. The CTO shares insight he has gathered about
one of the new startup competitors, he states that they are practicing the Agile
methodology, and this has made them nimbler, which has in turn allowed them to better
meet customer’s needs.
34
Breakdown of Orca’s Problems:
1.
2.
3.
4.
Learning Point(s):
35
System of Profound Knowledge
The W. Edwards Deming Institute - The Deming System of Profound Knowledge - 2009 –
http://deming.
One of the first places any company should start investigating how to make
improvements is by doing a Value Stream Mapping of all of their current delivery processes
in order to have a full “Appreciation of the System” (as per W. Edwards Deming’s System
of Profound Knowledge), the reason we do this is in order for us to always clearly
understand all stages of the delivery pipeline, by doing this we can identify the constraints
and dependencies that are decreasing the flow of deliveries.
It’s critical that organizations truly understand what an actual Delivery System is and
is not. One of the issues in most organizations, deals with silos not seeing the whole
Delivery Process, just their own slice of it. Examples of this are, an Agile process that
relies on End Users to provide complete requirements in the form of documentation before
36
the development of an application begins is not a Delivery system. The traditional software
development process where we create an application, but then leave the code in the
Version Control repository waiting for the QA team to start testing is not a Delivery system.
Or the traditional development process that tests an application but includes no planning
for installing it to a User Acceptance Testing environment or Production deployment is not
a Delivery system. An actual Delivery system starts when minimum requirements are
identified, are built in Development, and then transitioned into Production to provide
immediate value to the customer, and then continuously improved on.
Eliyahu M. Goldratt – Theory of Constraints (5 focusing steps)
Goldratt, Eliyahu M.; Jeff Cox. The Goal: A Process of Ongoing Improvement. Great Barrington, MA.:
North River Press
An issue that must be brought up when diving into Deming’s and Goldratt’s influence
on Agile and DevOps is the simple but sad truth, few if any people discussing Agile and
37
DevOps, pushing for them or leading the movements at their organization have ever read
any of Deming’s or Goldratt’s books. It is always interesting to listen to a DevOps
presentation from someone who gets the jist of what DevOps is truly about and then watch
as they discuss Deming’s 14 points or Goldratt’s Theory of Constraints, you could best
describe it as being a vast echo chamber at this point in the presentation. When the
presentation moves to discussing specific tools the audience always perks up and is
usually frantically taking notes. This is so unfortunate that these individuals will not take
the time to understand the true basis of what Agile and DevOps are about and instead
will continue believing that it's about the tools and there is some magical tool set that is
going to solve all their problems. You can almost always spot the person who does
understand DevOps, they are continually talking about TOC or Deming, and in many
cases, these people won’t stop talking about it.
Before diving into Goldratt’s Theory of Constraints, it's important to understand that
much of Goldratt’s work is based on the earlier work of Dr. Deming and that in many
cases Goldratt’s Theory of Constraints and Deming’s System of Profound Knowledge go
hand in hand.
To understand how the Theory of Constraints can be applied to DevOps, let’s take a
look at one example. It is not uncommon in most organizations for there to be a limited
number of resources, these can determine the output of the delivery system. In the Theory
of Constraints, these finite resources then become our constraints. What you discover is
that organizations attempt to design systems to manage the flow of work to and through
the constraints. Unfortunately, what we discover is that in many cases, the constraints
are actually being wasted and most likely underutilized.
If you take a look inside most organizations, you will see that often there are a few key
people who are needed for almost all projects, and these people are viewed as being almost
impossible to replace. These key people are seemingly always busy helping others and
rescuing the organization. The problem is that these key people are the constraints in the
organization. The critical problem is that if you closely obverse these key people, what you
discover is that they are often busy working on non-important tasks that have nothing to
do with adding value to the business. In terms of the Theory of Constraints, these
resources are not working to their max capacity, but at the same time are the busiest
people in the organization.
Goldratt’s Theory of Constraints and Five Focusing steps helps us overcome these
constraints by following the steps below.
38
Step 1: Identify the Constraint. One way in which we can pinpoint the constraint is
through the use of a Kanban board. We now can identify the constraint, so that we can
take steps to maximize the capacity of the constraint. In order to maximize the constraints
capacity, we need to make sure the constraint is only performing value added work. This
simply means we have to protect the constraint. In order to protect the constraint, we
need to make sure the constraint is only working on value added work that improves the
process and not time-wasting activities.
Step 2: In order to ensure our constraint is performing value added work, we need to
throttle the release of work to the constraint. Its normal for a constraint to perform work
on multiple projects and wear many hats. What we need to do is actually reduce the
number of work projects where the constraint is needed. It’s important to investigate if
the constraint is really needed on all the projects who are requesting time from the
constraint. We can do a couple of things here to relieve the constraint, we could look at
creating documentation or automating the constraint to relieve the burden.
Step 3: Subordinate the Constraint. We should now be seeing a constraint that is less
constraining on the system. It is important to work to keep it this way. Resist the urge to
force additional work through the system because this will not increase throughput,
instead we will just cause the constraint to become a bottleneck again. If needed, we
should build buffers or slack into the system to compensate for unplanned spikes in work,
so that we do not cause bottlenecks.
Step 4: Automate to Eliminate the Constraint. Even after completing step 3, the constraint
can still become a constraint to the system under certain conditions. To hopefully
permanently eliminate the constraint, we need to automate it.
Step 5: Repeat. At this point you have worked to improve the constraint as much as
possible. Now you want to see if the constraint has moved, so we start back at step 1.
39
Lean Manufacturing as a guide
Many of the leading DevOps practitioners will often repeat that in order to fully
understand DevOps and its implications, we must look historically at the consequences
of the Lean Manufacturing revolution of the 1980’s. The truth is that this should be a
frightening reading for the management at most companies. Because many US and
European companies were slow to adopt Lean Manufacturing, the result was that many
U.S. and European companies lagged so far behind in productivity and quality that they
were no longer competitive.
So then, what exactly is this Lean revolution, a good definition comes from the Eaton
Corporation:
“Lean is a Business philosophy that continuously shortens the time between customer
order and shipment by eliminating anything that increases the cost of production and
time of delivery to customers.” – (Eaton Corporation)
While looking at the Lean Manufacturing revolution we need to look specifically at the
contributions from the Toyota Production System, considered the gold standard for Lean.
There is a great importance in looking at TPS (Toyota Production System), you will find
that many terms being thrown around by the DevOps community originated at TPS.
Andon
In manufacturing plants this is normally a form of visual control that displays the
40
current state of work (Examples are abnormal conditions, work instructions, and job
progress information). Andon is one of the principal tools of Jidoka.
In manufacturing, Andon is commonly seen in the form of electrical boards which light
up to show at a glance the current state of work operations. An Andon board is critical
because it allows for speedy corrective action to be taken by when a problem arises. The
concept of Andon has been effectively incorporated by Amazon.
Cycle Time
This is the total amount of time it takes for a worker to complete one cycle of their entire
job process.
Jidoka
Jidoka, along with Just-In-Time, are two of the main pillars of the Toyota Production
System. Jidoka or Autonomation means "intelligent automation" or "humanized
automation". It refers to the ability of production lines to be stopped in the event of such
problems as equipment malfunctions, quality problems or work being late either using
machines which have the ability to sense abnormalities or using workers who push a line-
stop button. This prevents passing on defects. Recurrence prevention becomes simpler as
abnormalities become more obvious making it possible to “build in quality at the
production process”. This concept is critically important for DevOps leaders to
understand, since defects are prevented automatically, inspectors become unnecessary,
which in turn results in significant labor savings. This ties directly into shifting testing
left.
Kaizen
In the world of Kaizen, Muda (non-value added waste) exist everywhere and is related to
people, material and facilities, or the production set-up itself. Kaizen refers to the series
of activities whereby instances of Muda are eliminated one by one at minimal cost, by
workers pooling their wisdom and increasing efficiency in a timely manner through tools
such as Value Stream Mapping and the Theory of Constraints. Kaizen activities typically
emphasize manual work operations rather than equipment. Also, Kaizen is not an activity
to be performed by a specialist but can be – in fact should be – performed by all employees
at each job site.
Kanban
In manufacturing, Kanban is typically a small signboard that is the key control tool for
Just-in-Time production.
41
Kanban serves:
Instruction for production and conveyance

As a visual control tool
To control against over production
To identify irregular processing speeds
A tool to perform kaizen
Takt Time
Takt-Time is the time which should be taken to produce one widget. It is calculated as
follows: Takt Time = Total Daily Operating Time/Total Daily Production
Requirement*Daily total operating time is figure on the basis of all machinery operating
at 100% efficiency during regular working hours.
It's important to understand that these terms are not new and that the concepts they
represent have been scientifically proven out over a number of years. Often you will hear
individuals discussing DevOps and someone will make the comment that it's some new
methodology or way of doing things, in truth it's just applying tried and proven business
practices and concepts.
42
Bernoulli's Principle
Danielis Bernoulli Hydrodynamica, sive De viribus et motibus fluidorum commentarii. Opus academicum
ab auctore, dum Petropoli ageret, congestumJohannis Reinholdi Dulseckeri, 1738
Another scientifically based principle that is often discussed in DevOps is Bernoulli’s

principle for how the speed of a fluid relates to the pressure of the fluid inside a pipeline.
The principle states that within a horizontal water pipe that changes diameter, regions
where the water is moving fast will be under less pressure than regions where the water
is moving slow.
Bernoulli’s principle applies to Agile and DevOps when we are discussing the concept
of releasing code in small increments. In software development, this principle holds true
in that if we are releasing small increments of code through our delivery pipeline then we
can release them much faster due the decrease in pressure versus the traditional way
where we attempt push through code that has been piling up for weeks or months through
the pipeline at once.
43
Orca Evaluates Their Choices
For the next two days, much discussion and debate follow the presentation of all the
possible solutions. During this time one solution appears to have won almost universal
support, while another has bitterly divided the management team. The suggestion to adopt
the Agile methodology has won almost complete support from the entire management
team and has been delegated to the CTO to work out the implementation details.
A large battle ensues regarding the possibility of offshoring some of the development
and quality assurance work. On one side is the CTO who believes this is not a sound
solution for many reasons and the business side who issues dire warnings that Orca must
further cut costs in order to remain competitive. The CTO makes his case that his past
experience with offshoring has shown him that the work completed by off shore teams is
often not up to acceptable quality standards and is almost always delivered late. The
biggest objection the CTO makes regarding offshoring is the timing, he argues that
attempting to implement the Agile methodology while at the same time implementing
offshoring is a recipe for disaster.
The business side counters these arguments by stating that customer feedback and
surveys have shown that the current products and services are not perceived as being of
high quality or meeting their needs. The business believes by off shoring the quality will
be as good or better than the current quality, but that now they will have the ability to
scale up capacity when needed to get more new products/services/features out to the
customers, while at the same time dramatically cutting costs.
At this point the meeting dissolves into a finger pointing session, the business candidly
argues that the IT group is the one dragging the company down since they are unable to
support the business requests in a timely manner or with acceptable quality. They further
state that it takes a couple of tries just to get each release out to the customers and that
they then spend weeks fixing bugs and adjusting the product to meet the customer’s
needs. The business at this point has little to no trust in the IT to support them, even
though they believe that they provide them with the necessary information to fulfill the
customer’s needs.
The CEO finally cuts off the arguing and suggests that this is the kind of healthy
discussions they have needed in order to flush out Orca’s problems. The CEO after some
deliberation decides that while he also believes that offshoring is the way to go, he will
44
also heed his CTO’s concerns and only allow the off shoring to proceed with a pilot project
which will be determined by the CTO and the business side.
Breakdown of Orca’s Choices:

1.
2.
3.
4
45
Learning Point(s):
Conflict Creates Failure
One of the interesting facts of Agile and DevOps is that it requires organizations to
untangle the incentive process they have normally put into place over the years, because
this incentive process is actually driving many of the cultural issues that are preventing
the employees from making progress in the drive to increase speed to market and improve
quality. So how you ask is the incentive program causing many of the cultural issues
inside most organizations, first if we look at the natural way in which an Operations group
tends to look at the world, it's through what is termed a “Horizontal Optimized world” and
our development teams have what is termed a “Vertical Optimized worldview. In the
Horizontal Optimized view, the Operations team looks at IT as being something that has
a limited set of resources that need to be carefully pieced together in order to guarantee
efficiency, maximize existing resources and is stable. At the same time, our Development
teams are looking at IT in a Vertical Optimized view where the only considerations are
46
how fast can they deliver new software to the end user. These two differing views have
created what is termed the “blame game” in many organizations because the development
team is simply concerned and more importantly incentivized to get new software out as
fast as possible, while in the Operations side they are concerned with and incentivized to
make sure the software is stable, has acceptable uptime, is secure and that the
infrastructure is optimized.
Organizations must understand these two inherit competing views and act to align the
views of both groups by changing the incentive structure to help drive a common purpose
by both Operations and Development. If organizations do not change the incentive
structure to align both groups, then they miss the opportunity to use methodologies such
as the Theory of Constraints to tackle the issues that are truly impeding their progress in
improving speed and quality. The reason for this is that the Operations team will
continually be seen as the roadblock by the entire organization in their drive to increase
the speed of delivery and improve quality. The truth be told, the Operations team really
isn’t the roadblock, the roadblock (constraints) are really the entire delivery process itself.
By aligning incentives, we bring together Operations and Development to take a look at

the delivery process and to determine the actual constraints that are impeding the
progress. A few interesting things happen when the incentive process is aligned, the
Operations team will become more open to changing the infrastructure to allow for faster
deployments because they find that delivery pipeline that gets put into place becomes
much more transparent, allowing them to have an actual view into the upstream process
in order to better understand the code that is coming their way, how it's being tested,
what the code actually does and how it could affect production.
47
Technical Debt
Rawsthorne, Dan. “So What Is Technical Debt in Scrum Anyways?” 3Back Scrum & Agile Blog, 22 Dec.
2014, blog.3back.com/scrum-questions/so-what-is-technical-debt-in-scrum-anyways/.
Erickson, Aaron. “Informit.” Don't "Enron" Your Software Project | InformIT, 10 Oct. 2009,
www.informit.com/articles/article.aspx?p=1401640.
No discussion of Agile or DevOps should ever take place unless the organization
interested in adopting them truly understand the implications of Technical Debt. Research
over the years has consistently shown that organizations across the board are spending
anywhere from 70 to 80 percent of their development budgets to simply maintain their
existing code. Unfortunately, when these numbers are presented to the majority of
organizations they will scoff at these numbers and say that this isn’t true at their
organization because they are always putting out new software. The problem is that these
companies and in particular the business side do not understand that all this new
software they are putting out is actually fixes for existing software. In organization after
organization when you truly dig down into the development work being done what you
find is that they will fire fight the same issue over and over and never tackle the underlying
issue.
48
Once an organization fully grasps the Technical Debt issues they most likely face, they
will find that the idea of Agile and DevOps becomes much easier to swallow and most
likely will gain organization wide support. Gaining a perspective on the Technical Debt
facing an organization isn’t that hard to discover, tools such as SonarQube do a very good
job of painting a picture that anyone in the organization can understand. A key point to
make to the business when discussing why the Technical Debt issue must be tackled is
explaining in simple terms what the actual financial purpose of creating software is for
the company. When we create new software products or features we absorb costs for the
resources to develop the software, test the software, deploy the software, the
infrastructure to run the software and maintenance of the software. The only way in which
an organization receives an acceptable ROI on this effort is if the software has a long
enough life span to pay back an acceptable ROI for the organization. What many
organizations and individuals are failing to understand is that Technical Debt in many
situations is shortening the lifespan of the software they have created, sometime
dramatically shortening it and the organization is never fully realizing the full ROI they
should be.
Orca Makes a choice
The CTO of Orca decides to call a meeting of his IT management team to inform them
of the decision’s that the executive management team has decided on as the path forward
for Orca. The team is understandable shocked to hear about the decision to start off
shoring the development and quality assurance, almost immediately some managers start
thinking about what this means to them in terms of job security. The immediate feedback
to the CTO is one of fear, push back and complete lack of support.
The topic of the meeting next turns to the decision to implement the Agile methodology,
which also receives lukewarm support from the management team. The reason for the
lack of support for Agile runs the gamut of reasons, some are opposed to change in
principle, some managers are familiar with Agile and wholeheartedly believe if it was done
correctly would be great for Orca but are sure it will follow the path of previous
improvement attempts. One of the underlying fears among all the management team that
49
is familiar with the concept of Agile is that it will erode their current authority and make
them expendable.
The CTO assigns two of his senior managers along with himself to plan out how exactly
they will move forward with implementing Agile at Orca. The team gets together and
initially starts hashing out ideas for how to get the implementation rolling and in addition
planning for risks and determining strategies for countering these risks. There is almost
immediate agreement that the rollout needs to incorporate outside resources due to the
built-in skepticism that exists inside the IT group to new methodologies, due to all the
previous failures. The team does some initial research and decides to interview three
different Agile consulting firms to hopefully lead the initial training and provide follow on
coaching as needed.
The three companies are contacted and over the next week some initial discussions
follow for the three companies to flush out what exactly Orca is looking for and also
attempt to determine what is currently happening at Orca. After the initial discussions,
dates and times are set up for the representatives of the three companies to come in and
give their pitch to the Orca team. The first firm that comes in to give their pitch highlights
that they can implement any of the various Agile methodologies along with working
directly with the management team to help foster the cultural change that is required in
order to effectively implement Agile. The firm also explains that they will come in initially
and do an assessment of Orca and then leverage that to help drive the changes necessary
at Orca to truly see the benefits of Agile.
The second firm also comes in and for all purposes give the same plan to the Orca team
as the first firm. Both companies emphasize that in addition to rolling out the Agile
methodology in one of its forms, to truly be successful Orca must make the corresponding
cultural and tooling/process changes required to see the benefits of Agile.
The last firm to come in is a large multinational firm that explains that they can come
in and not only roll out one of the Agile methodologies through a pilot project, but also
train individuals at Orca to eventually take over and run the Agile rollout across the IT
group.
The CTO gets together with his two senior managers and they discuss the presentations
they have heard and try to break down what the three firms offer and how this would work
best for them. One of the first discussions they have is the idea presented by the first two
firms that Orca will need to make changes to their culture, all three managers agree that
50
they believe Orca has a good culture and that this is not really something that is needed
at Orca. While they realize, there is going to be initial skepticism about the Agile rollout,
if they support it and do it right it will catch on with their employees. The managers also
discuss the fact that the third firm’s plan is to do some initial coaching and training and
then the teams at Orca would take over. All three managers agree this would be the ideal
scenario, since they believe they just need some initial help getting started and in addition
this would also be much cheaper than what it would cost if they went with either of the
first two firms. The decision is made, and dates are arranged for the third form to kick off
the move towards Agile at Orca.
Breakdown of Orca’s Choice:

1.
2.
3.
4.
51
Learning Point(s):
Deming’s Deadly Diseases of Management
Deming, W E. Out of the Crisis. Cambridge, Mass: Massachusetts Institute of Technology, Center
for Advanced Engineering Study, 1986. Print.
An almost constant refrain heard in many software shops by employees is the complaint
against management for their continuously changing focus on today's current critical
issue. What the employees mean by this is that they are constantly being pulled off
projects to concentrate on another project that management has deemed to be more
critical on that particular day. Or another common complaint is that teams will be directed
to start automating the testing of an application under development and after they get
started they will be pulled off to help another team manually test an application that has
suddenly become more critical.
The problem with this lack of focus ties directly into Deming’s lack of constancy of
purpose by management and Feedback Loops in DevOps, when team members are
52
continually being shifted around by management to focus their efforts on different
applications or testing they will never fully gain the critical information gathered by
Feedback Loops and thus will never be able to apply this missed opportunity for
improvement.
Another issue that DevOps practitioners call out that Deming also identified is the need
for management to stop the traditional way of evaluating employee performance.
Management must learn that they will never see improvement in performance that results
in increased speed of delivery with corresponding quality improvement unless they start
authorizing their lower tier management to rewards employees on the spot who exhibit
behaviors that encourage process improvement, experimentation with new process and
tools and most importantly teams and individuals taking ownership of these new tools
and processes and seeing them through to completion.
Another issue that Deming called out and is discussed by DevOps practitioners is the
need to re-evaluate the metrics relied upon by management. An example of this is the
focus on velocity by teams practicing the Scrum methodology. If you discuss progress with
the business side or IT management who are using the Scrum methodology they will
continually emphasize and rely upon the team velocity numbers to determine how they
are progressing and to measure performance. If you then turn this around and ask these
same individuals the rate of waste for the stories adopted by the teams, they will most
likely look at you with a bewildered look on their face. The reason for this is that no one
inside the company has thought to try and track how much waste was worked on over a
given period of time by the teams. By waste we mean how many stories were taken in by
teams and worked over a course of time that were possibly never finished or were finished
but never made it to production or made it to production but turned out to not meet the
end user's needs.
53
Choosing an Agile Methodology
A lesson that organizations all too often discover the hard way is that the various Agile
methodologies are not a one size fits all. Many organizations have taken the follow the
leader mentality when it comes to adopting an Agile methodology. For several years, it
seemed that the Scrum methodology was all the rage and every organization thought this
was the only way to do Agile. Over the last couple of years Kanban has started to become
the new rage and organizations have either made the wholesale switch over to Kanban or
organizations just starting out with Agile have decided Kanban is the way to go with all
their teams.
54
The reality is that the various Agile methodologies are not a one size fits all in every
situation and companies should evaluate the various methodologies for each team. An
example would be some of the new “Platform” teams that have come about recently with
DevOps and who see the internal Applications teams as their customers, and who they
are building out delivery pipelines for. These Platform teams may be better suited to using
Kanban due to the nature of DevOps work which can often be trial and error. The existing
application teams may in turn be better suited to using the Scrum methodology since they
have a better handle on what the application they are building entails.
Dealing with the fear of change
When we discuss Agile and DevOps, we can simply not get around the subject of
cultural change and specifically the fear of change it often brings about. Any individuals
who wish to help undertake the implementation of Agile and DevOps inside their
organization should be aware of the challenges that will occur due to the fear of change.
If you ask many managers inside an organization who are seeking to move towards
DevOps how they will handle the cultural changes that must take place inside their
company they will almost to a person state that they believe their culture has no issues
and that Agile and DevOps will be a success. The truth is that most of these individuals
are completely oblivious to the cultural rot that has been taking place over the years inside
their organization.
55
The first concern anyone attempting Agile and DevOps should be concerned with is how
are they going to handle and neutralize the Laggards that will attempt to sabotage the
movement at every turn. The first thing to understand about the Laggards is that they can
include individuals at every level. The Laggards will fear Agile and DevOps for a number
of reasons, it may be seen at a threat to their authority, they may believe it could uncover
their lack of skill/abilities, they may simply just be against change of any kind or they
may believe it could led to the loss of their position. The identification of the Laggards is
in most cases is not that difficult, they more times than not will be fairly vocal in their
resolve against Agile and DevOps or any other change. It is important to also understand
a few of the key ways these individuals will attempt to sabotage Agile and DevOps, one
way is that they will go to the business side and sound alarms if topics such as Unit
testing are brought up. The way in which they will attempt to frighten the business side
is by falsely stating that if Unit testing is put into place the development teams will have
a decrease in velocity and the business will no longer be able to get as much new work
pushed through the pipeline to the end users. This fear of loss of velocity to the business
must be quickly countered with the truth, yes there may be a decrease in velocity at first,
but this will be more than made up for over time.
Another false argument that the Laggards will make is to try and tie Unit testing and
Test-Driven Development together. The Laggards will almost certainly bring up articles by
some individuals in the software industry who argue that Test Driven Development isn’t
affective. This red herring argument must be explained to everyone involved, Test Driven
Development is just one methodology for doing Unit testing and few if any in the industry
would argue that Unit testing shouldn’t be performed.
Possibly the number one argument that the Laggards will make in their case against
Agile and DevOps is that their organization is “unique “and that Agile and DevOps will not
work for them. This is an absolutely false argument and one in which the individuals
working towards Agile and DevOps must counter with the truth, their company is not
unique and faces almost the identical set of challenges from a cultural perspective that
almost every other organization does. The only real difference may be the tool stack that
is in place at these organizations.
Lastly, it should be made clear that if the Laggards are defeated in their opposition to
Agile and DevOps, the vast majority of them will end up leaving the company on their own
accord or they will finally be pushed out by management who finally sees the light and
realizes the what obstructionist this group really is. Make no mistake, it is often very
56
difficult to get management to realize that these Laggards are a cancer inside the
organization.
Now that we understand that we need to identify and neutralize the Laggards, lets looks
at the makeup of the rest of the organization. The majority of individuals inside an
organization will fall into a group we call the “Followers”. The Followers will generally
follow whichever group they feel is winning the Agile and DevOps movement. Make no
mistake, a number of these Followers will be inside the Laggards camp for strictly political
reasons. The Followers inside the Laggards camp will stick with the Laggards until the
very end when either the Laggards win or the forces moving towards Agile and DevOps
win.
Finally, we have what are termed the “Innovators” who are individuals inside an
organization who can see that there has got to be a better process then what is in place
currently. These Innovators can often be identified by attending various Meetups on new
tools or methodologies. The Innovators will most often take time out of their personal
schedule to try and learn about new tools and methodologies and often bring back these
ideas to the organization. When the Innovators inside an organization are identified and
start attempting to spread the awareness of Agile and DevOps, it's particularly critical for
management to first, reward these individuals for their efforts so that everyone can see
this and to also help protect them from the Laggards, especially if one of the Laggards has
any type of supervisory over the Innovator. All organizations must also be cognizant that
historically the individuals who most often make up the Innovators have not been as
valued by organizations as much as individuals who are seen as being good at organizing
and managing. This means that organizations must take a hard look at their evaluation
process and determine of they are valuing the correct skills in their employees.
57
The Implementation begins
After the decision is made to go with the large multinational firm, a series of meetings
is scheduled and the details of the Agile pilot project are hammered out. The decision is
made to start with one of the web development teams since they will be starting a new
project in the next few weeks. During these meetings to hammer out the details, it is
decided that Orca will go with the Scrum methodology since they have a number of
employees who are at least a bit familiar with this methodology or they have used it at
previous employers.
The lead trainer for the consulting firm sets up the initial training sessions for the web
development team. The two-day training session for the team is given and is basically
uneventful. There is no real push back by any of the employees at the trainings session
and the trainer and managers feel good about the move forward with the Agile push for
the web team. At the same time as the training, a consultant works with the designated
product owners to go over story and acceptance criteria creation. An initial set of stories’
is created for the web development team and they are put into the newly created backlog.
The lead trainer gathers the web development team for their very first sprint planning
session. The trainer again explains how story pointing works and covers how the team
will then break down each story into smaller chunks to be worked on. The first story is
read and the team using poker cards lays down their cards. Almost the whole team agrees
on the points for the story except for one team member. The trainer now explains that
this individual can explain how they came up with this number and then the team will re-
point and hopefully come to agreement. After explaining their reasoning for pointing the
story the way they did, the trainer asks if anyone has questions for this individual and
nobody does. The team re-points and the entire team points to the higher number that
the lone team member did in the first round of pointing. The trainer reads the next story
and the team is again asked to point the story. Just as in the first story, the same team
member points the story higher than every other team member. The individual once again
explains their reasoning and the team once again changes their points to that of the
individual. At this point the trainer recognizes an issue they have experienced before and
attempts to explain to the team the importance of everyone voting their own mind.
After the story pointing sessions, the trainer meets with the CTO and his two senior
managers to explain the cultural issue that they observed during the pointing session.
The trainer explains that they have experienced this issue with other clients and that the
58
managers must quickly intervene with the team to resolve this issue. The trainer explains
that in certain cultures, individuals will defer to others in their culture even when that
other individual is not senior or does not have a higher skill level. The managers listen to
the trainer and agree that they will have a talk with the team member. The managers have
a discussion with the team member, but do not broach the cultural subject with them. As
the sprint continues, these cultural issues continue to affect the team.
As the weeks go by, other teams are given initial training and start their journey forward
with Agile. Eventually all teams at Orca are practicing some form of Agile and the training
is completed by the consulting firm for Orca to take over the Agile efforts. As the teams
move forward with Agile, the CTO works with the business side to iron out an agreement
for starting the off-shoring efforts. Finally, each side comes to an agreement on a project
to send to the offshore team that has been selected. The CTO feels relief that he has been
able to get the business side to agree to offshore a project that does not have a high
strategic importance to Orca.
As the weeks go by the executive team starts closely monitoring the Agile efforts to
hopefully see the results they have been hoping for in terms of quality and speed to
market. Unfortunately, the management team is not seeing the results they had hoped for
and had also planned for. At this point the executive management team asks the CTO and
his team to start closely monitoring the teams to determine why they are not seeing the
expected results. The CTO begins asking each of his managers to have their subordinates
start tracking and reporting their work each day, so that they can hopefully determine
why they are not seeing the results they expected. The managers at this point start hearing
feedback from their subordinates about having to spend the additional time tracking their
work and doing this extra reporting.
At the same time, the CTO starts meeting with key individuals in the various
departments to try and gain insight into what needs to be done. The CTO hears the same
message from individual after individual, they can’t get the code loaded into environments
in a timely manner, merging is a nightmare, they don’t have time to thoroughly test and
the business keeps making changes in the middle of sprints. At this point, the CTO decides
to bring in an industry expert to assess the IT group at Orca and hopefully help Orca start
seeing the benefits they not only had hoped for, but desperately need.
The industry expert comes in almost immediately and starts a series of interviews with
individuals across all areas of IT and the business side. After the interviews are wrapped
up and an assessment is put together, the expert presents the findings to the CTO and
59
his management team. The findings not only point out what the two consulting firms not
chosen for the Agile implementation pointed out about culture change, the findings also
discuss major issues Orca has with their Agile implementation and the desperate need to
implement the corresponding tools and processes to support Agile. In a nutshell, the
industry expert explains that Orca is not only not doing the upstream Agile correctly, but
they have almost no downstream tools or processes in place to take advantage of Agile
even if it was working effectively. The industry expert advises that as a starting point, Orca
should start implementing the tools and processes needed to try and get Continuous
Integration in place and if they are successful with that, then attempt to look at
Continuous Delivery/Deployment and the gold standard DevOps.
Breakdown of Orca’s Implementation:

1.
2.
3.
4.
60
Learning Point(s):
Changing the Culture
“Gartner Highlights Five Key Steps to Delivering an Agile I&O Culture.” Gartner, 20 Apr. 2015,
www.gartner.com/newsroom/id/3032517.
Recent industry research has shown that up to 90 percent of organizations will fail in
their quest to move to DevOps. The reason for this is very simple, DevOps will only work
when both Dev and Ops are willing to collaborate. An organization must be willing to
embrace change at a behavioral level to initiate and achieve culture change. If not, then
the DevOps initiative will fail. The reasons why organizations have not been willing, and
most are still not willing to embrace change at the behavioral level are many, but we will
endeavor to discuss a few.
We will start our discussion with management, the issues with management are
unfortunately chronic in the software industry. Across the majority of software
organizations, a trend has developed where the best and brightest individuals who are
also culturally and behaviorally suited for management have not been promoted to
management positions, instead the trend has been for individuals who are better at
playing internal politics to be promoted across the industry. The reasons for this are many,
but they boil down to a few common threads across organizations. First, the lack of
61
knowledge of Deming’s teachings, Lean principles, and basic business methods by top
management is almost shocking. If top management is unaware of these basic teachings
and principles, then how can they ever be expected to identify and select the top
management prospects coming up through the ranks. The answer is that they don’t
identify the top candidates and instead promote individuals who are often completely
lacking in the business, cultural and interpersonal skills required to perform their duties.
What this leads to is an organization that is not only unwilling to change at the behavioral
level, but is simply unable to due to the individuals leading the organization.
Another issue that plagues management in software organizations is the lack of cross
cultural communication knowledge. The reality today is that most individuals in software
development come from a broad range of countries/cultures and few if any organizations
actually provide any type of training or discussions in regard to cross cultural
communications. All too often you will come across organizations who promote individuals
who are technically strong but come from a different country/culture then where they are
based and time and time again they will lack the basic understanding of cross cultural
communications and this will lead to a butting of heads with others within the
organization.
One organization overcame this issue by putting into place a management education
policy as follows, any individual who is hired or promoted to a management position inside
this organization must either hold a Masters of Business Administration degree or
acceptable equivalent or if they do not hold an MBA they will be required to sign up in the
Executive MBA program that this organization participated in at the local University. The
organization that put this policy into place did not believe that getting an MBA was
necessarily the answer to selecting all the correct management candidates, instead they
looked at it as helping highlight the best candidates in three ways. The first way they
thought it helped was by knowing that most accredited MBA programs required a number
of team projects, this was believed to help ingrain working collaboratively in the new
managers. The organization was also aware that the MBA program they participated in
also required all participants to take a cross cultural communication class, which was
thought to make the managers more culturally sensitive. Lastly, it was thought that
anyone who would undertake getting their MBA and finish, especially if they were also
working, would also be someone who would take on initiatives/projects and see them
through to completion.
Unfortunately, the cultural and behavioral issues are not confined to management
alone, the lack of management direction has allowed cultural and behavioral problems to
62
fester below them. One of the chief problems that is seen in almost every organization is
what has been termed the “Hero Cult”. The hero cult issue normally materializes in the
form of developers who have been with the organization for a number of years. The
developers who display hero cult personas generally are seen by the organization as having
vast knowledge of an area of the application, framework or toolsets and often come to the
rescue of the organization in times of crisis. The problem is that if the organization would
just look under the covers at these developers, they would find that in many instances
these developers created the crisis in the first place by either having complete lack of
documentation for their work, or they had poorly written code or they choose
tooling/frameworks that didn’t conform to what the organization actually needed. Often
times, others in the organization are keenly aware of the problems created by these hero
cult individuals, but due to the high regard management often holds them (or fear of losing
them) others will not speak up or argue against these individuals.
Another cultural issue that often shows its ugly head in organizations is the concept of
“Shadow Responsibilities”. One area where shadow responsibilities often shows up in the
area of Performance testing. Teams will often need performance testing of the work they
have completed in their iterations, but when they reach out to get access to perform some
performance testing they discover that this falls under the realm of some manager, group
or individual in the organization and that in order to get the performance testing they
must jump through a number of hoops and in most cases, must rely on someone else to
perform this testing.
Another common problem that often appears when organizations are attempting to
move towards automated testing is the problem of “Emphasis on Titles”. All too often
management will select a testing Lead or senior tester to help lead this effort, but as is
often the case, this employee lacks the technical knowledge and even worse shows little
interest in automated testing and in some instance, will even publicly downplay the need
for automated testing. Often times, its younger newer employees who have the technical
skills as well as enthusiasm to help lead these efforts.
The movement towards Agile and DevOps requires the adoption of critical teachings of
Lean, Deming and traditional business practices which successful Agile and DevOps
adopters have realized helps change the cultural and behavioral dynamics inside an
organization. These changes to the dynamics inside an organization help root out many
of the cultural and behavioral issues discussed above.
63
Management by Objectives
Drucker, Peter F. The Practice of Management: A Study of the Most Important Function in America
Society. Harper & Brothers, 1954.
Management by objectives was first popularized by Peter Drucker the most revered guru
of American management in his 1954 book The Practice of Management. Management by
objectives is the process of defining specific objectives within an organization that
management can convey to organization members, then deciding on how to achieve each
objective in sequence. This process is supposed to allow managers to take work that needs
to be done one step at a time to allow for a calm, yet productive work environment.
Dr. Deming wrote in his 14 points that he thought MBO was a bad idea and he was
emphatically clear about this. About 15 years later, Drucker himself came around to the
same point of view and recognized that MBO had failed. So, what did Deming find wrong
with it and why does DevOps also discourage it?
Management by Objectives asserts that it is essential for managers to have quantitative

objectives to work towards. They should set these objectives themselves, consistent with
the objectives of the business as a whole. But Management by Objectives is short on how
this can actually be done, how should you identify proper metrics, how you set targets for
64
each metric, and how you go about reaching these targets. It is not an accidental omission.
In the Management by Objectives world, managers are supposed to be able to find the
answers themselves. It is part of their job, and they are supposed to have the required
talents to do so.
Dr. Deming instead saw arbitrary objectives, unrelated to the purpose of the business,
set without any plan on how to achieve them or even a means of taking accurate
measurements. Deming sees MBO as “an attempt to manage without knowledge of what
to do.” The managers focus on outcomes without looking into the processes (DevOps
specifies that we look at processes via methods such as The Theory of Constraints) that
produce them. Managers end up gaming metrics rather than improving processes and
playing management firefighting.
So why do so many software organizations still lean on Management by Objectives and

refuse to look instead at improving processes as Deming and DevOps suggest? The
problem lies in the previous section where we discussed cultural change, Management by
Objectives has become ingrained in the culture of most organizations and because
management is so unfamiliar with Deming and Lean they never question the continued
use of MBO or even realized they are using it. Basically, because so many managers lack
basic business acumen they continue to follow the MBO play book that they have been
exposed to.
65
Agile Upstream and Downstream
Some of the common topics in almost every DevOps book are what is DevOps, its history
and its relation to Agile. The common theme among most DevOps thinkers is that DevOps
is sort of a natural extension of Agile, in other words we are extending Agile across the
entire organization to include Operations. The reality is that DevOps is more likely a
reaction to Agile. What do we mean by a reaction to Agile, we mean that the Agile
methodology movement has failed across software development organizations. Why do we
say this and why is DevOps the reaction to this failure?
There are numerous reasons why we say the Agile methodology movement has failed,
we will discuss some of the main reasons. If you were to walk into the majority of software
66
organizations today and ask if they practice Agile, the definitive answer would be yes, and
they would then go on to explain how they do Sprints, have User Stories and have Daily
Standups. When you then counter that just because someone is doing Daily Standups,
writing User Stories and having Sprints doesn’t mean you are doing Agile, they have a
bewildered look on their face.
When you dig under the covers at these organizations who tell you they are doing Agile,
what you quickly discover is that they are actually doing their own interpretation of Agile.
You find that they are inserting User Stories during Sprints, their Daily Standups turn
into hour long sessions and they force an unmanageable number of Stories into Sprints
among other things.
When you ask the employees at these organizations if they believe they are doing Agile
you will often get a wry smile from them or some will even just come out and say no we
do some form of Scrum fall or they will say we don’t do anything remotely Agile.
So, how do you get this differing perspective on Agile inside an organization between
leadership and employees? This comes back to an issue we discussed in an earlier
learning point, if management doesn’t have an understand of the underpinnings of Agile
then there is almost no possibility that they will implement it properly. Instead what we
see in organizations is management cherry picking the parts of Agile that are easy to
implement and don’t cause cultural issues.
We also need to look at Agile Upstream and the natural progression to Agile
Downstream that the original founders’ thought would occur. The original Agile founders
thought that as organizations adopted Agile and started to mature, they would understand
that in order to fully incorporate the benefits of the Agile methodology they would need to
implement the changes needed for Agile Downstream.
One of the natural progressions the original Agile founders believed would occur over
time was that Quality Assurance and Programmers would all evolve into Developers. The
Agile founders realized that in order to have releasable software at the end of the short
Agile iterations, organizations would naturally conclude that the only way to do this was
to make the move from cumbersome manual testing to automated testing. In order to
make this move organizations would need Quality Assurance to morph into a more
technical role and programmers to play a larger role in testing. Unfortunately, neither of
these changes has occurred across the majority of organizations.
67
In addition, as Agile matured inside organizations the founders also believed that it
would become clear to all that in order to truly be Agile and release higher quality software
faster, these organizations would discover that this was an impossibility without
incorporating Continuous Integration and Continuous Delivery. DevOps has realized that
none of these changes required to fully become Agile has taken place in the software
industry and this is why DevOps has come about, it can be looked at as a more forceful
attempt to get organizations to make the changes necessary to take advantage of the Lean
principles that Agile is based on.
Orca Must Make Tough Decisions
At this point the CTO of Orca must make a number of extremely difficult decisions in
order to try and salvage the Agile implementation and show the executive management
team the results they were expecting from the move to Agile. The CTO decides that the IT
departments will be asked to champion several of the suggestions the industry expert has
made in order to try and move Orca towards Continuous Integration. The managers of the
development team are brought together and asked to have their departments champion
the Continuous Integration components that the industry expert has laid out for them.
The first department that is asked to champion one of the CI components is the
developer group, they are asked to come up with a plan to start putting into place the
habit of Unit testing in the Agile Sprint teams. Almost immediately the managers of the
development team feel a sense of dread knowing the challenges that have just been asked
of them. The development managers decide to present the Unit testing request to the
developers at their monthly staff meeting in two days. At the monthly staff meeting, the
development management team tries to cautiously broach the subject of Unit testing.
Immediately some of the senior developers balk at this suggestion, they quickly lay out
several reasons they believe make this a bad idea. The first issue presented by some of
the developers is that it's simply too expensive to have developers writing tests, they state
that developers are very costly, and they shouldn’t be spending time doing tasks that the
less costly quality assurance team should do. In fact, some of the developer’s state that
the best solution would be to hire additional testers. The developers also point out that
they currently don’t have enough time as it is in their sprints to complete the work they
68
are doing and that by adding Unit testing it will cut down on their current capacity, which
the business will never accept. The hour-long meeting turns into open revolt and the
management team at this point decides to regroup and try and sell the idea of Unit testing
in a different fashion. The day after the meeting, a couple of the developers send an email
to all of the developers, the development management team and CTO with an article
attached arguing that TDD (Test Driven Development) is a bad idea and that this proves
their point regarding Unit testing.
At the same time, the CTO has also chosen to attempt to roll out Automated Testing,
starting with Functional Automation. At the meeting to discuss the Automated Testing,
the Quality Assurance management team quickly jumps in and demands that Automated
testing falls under their domain and that they will be the ones to roll it out. A couple of
managers from development and operations interject that they believe to be successful
with Automated testing, it actually should be a joint effort since effective Automated
testing is actually coding. The Quality Assurance managers quickly cut of this argument
by pointing out that QA is responsible for quality and testing and therefore they must
have control of this initiative. The CTO at this point agrees that the QA management team
will roll out Automated testing. The Quality Assurance management team decides to bring
this initiative to the weekly QA meeting and hopefully put together a plan for moving
forward. At the weekly meeting, the initiative for Automated testing is presented, the QA
team has a number of questions, how will a tool be decided, will training be offered and
does the Orca management team understand that Automation should just be a
supplement to the manual testing that is required? At the meeting, a couple of QA team
members are selected to work with one of the QA managers to evaluate several tools and
then select the direction to go in. After reviewing several tools over the next couple of
weeks and talking with potential vendors, the QA tool selection team recommend that
vendor Alpha be brought in to build out the needed framework, automate a number of the
existing regression test cases and then eventually train the Orca QA team to take over.
The CTO also tasks the Operations team with putting together the infrastructure to run
Continuous Integration. The Operations team gets together to determine a plan for
building the CI infrastructure, at the meeting the team hashes out some ideas for how to
structure the CI, what tools should be used and the policies that need to be put into place.
One of the policies is that the CI engine, environments and other CI components will be
put behind a firewall and only the Op’s team will have access to the CI for
security/compliance reasons. The team also determines that a more comprehensive
Change Management system/tool will need to be put into place in order to better track
what is going through CI. Finally, the team discusses what comes after CI and touches on
69
CD and DevOps. A couple of Op’s members argue that Orca will never be able to move to
CD or DevOps because they are to unique in terms of their infrastructure and that DevOps
will not work in a company like theirs.
The management teams running the various initiatives report back to the CTO about
their progress and what they have learned, the CTO is left speechless when he learns
about the resistance to the initiatives, the direction they have chosen and the lack of
progress. The CTO decides to meet once again with the industry expert he hired to help
direct Orca in the right directions, the industry expert quickly shots down the arguments
presented by individuals and teams with facts backing up his recommendations and
argues that the CTO must change the culture in order to be successful. At the same time,
the CTO also discovers that one of his Senior managers he has been relying on has been
sabotaging his efforts to implement Agile and CI. The CTO is furious at this point, but also
unsure of how to proceed. After some time thinking about the problems that confront him,
he decides that he will hand pick some developers and Ops folks that he trusts to work
as a stealth team to start putting into place some of the CI, CD and possibly DevOps, and
once they are done he will impose this on the various Agile teams. The CTO sets about
trying figure out how to go about creating this stealth team, knowing that a couple of the
critical individuals he will need are so busy with every initiative at Orca that not only
getting any of their time will be difficult, but also not alerting everyone to what they are
working on will be difficult to near impossible.
At the same time, the CTO is attempting to drive the Agile and CI efforts the Business
side has gotten wind of the changes happening on the IT side and is simply not on board
with them. The business side still believes that all the issues facing Orca could be resolved
by moving more of the development offshore, since they believe this will increase capacity
and they also believe that if the development side did a better job quality wise they would
not be in the predicament they are now. The business believes that plenty of new features
and applications they have come up with are moving through the development process,
it’s just a matter of IT executing better. The business side finds that the development side
often does not grasp what they have asked for, even though they believe their old
requirements process gave the developers more than enough information to create the
new features and applications. The leaders of the business side are especially concerned
about attempts to include Unit testing in Sprints, since they have been warned by
individuals that this will lead to a reduction in velocity and that is the last thing they
believe they can afford right now.
70
Breakdown of Orca’s Dilemma:
1.
2.
3.
4.
Learning Point(s):
71
Why we Unit Test
One of the most scientifically proven quality improvement processes in software

development is also one of the least implemented. Research over a long span of time has
shown again and again that not only is Unit Testing one of the most effective quality
improvement processes, but also one of the most cost effective. If Unit testing has been
overwhelmingly proven to improve quality and greatly reduce costs, then why hasn’t the
industry adopted it across the board?
One of the first arguments you will consistently hear from management is that it’s not
effective or it is too time consuming. When you then question managers where they found
this evidence that it’s not effective or too time consuming they are never able to provide
any type of evidence, instead what you discover is that some of their developers
complained and the management never investigated to find out if the developers were
accurately depicting Unit Testing. As has been pointed out in previous sections, this is a
fault with management for not doing their due diligence in collecting information and
making scientifically based decisions as Deming would call for.
Another way that Unit testing gets torpedoed by Developers is through confusion with
Test Driven Development. Often when management comes to development about
implementing Unit Testing in the move towards Agile and DevOps, some developers will
72
come back to management with articles from certain well-known software industry names
who have written articles stating their belief that Test-Driven Development is not effective.
The problem here is that the articles by these individuals are not arguing against doing
Unit Testing, they are only stating that they don’t believe using the Test Driven
Development methodology is effective. It’s critical that the leaders of the Agile and DevOps
movement inside an organization be prepared to clarify that Test Driven Development is
just one method for doing Unit Testing and that there is no one in the software industry
arguing about the effectiveness of Unit Testing.
The other way in which Unit testing is all too often shot down inside an organization is
by convincing the Business side that Unit Testing will greatly reduce velocity of the
development teams. As soon as the idea of Unit Testing is brought up, it almost never fails
that some individuals on the development side will go to the Business side and attempt to
convince them that Unit Testing is going to greatly reduce velocity. The truth is that yes,
in the short term if an organization has not been practicing Unit Testing there will be an
initial ramp up period where velocity will most likely be reduced. What is not conveyed to
the Business is that over time this initial ramp up time will be paid back many times over
and maybe more importantly the increased quality will also result in a higher end user
satisfaction.
73
Breaking down silos
To understand the critical mistake being made by a number of organizations in regard

to implementing DevOps, we must look all the way back to the Waterfall methodology and
we will see the same mistake has plagued Agile. When we look at Waterfall we see clear
and distinct boundaries between departments that have their own silos. Inside these silos
we see departments that own their own tools and processes. The creation of silos results
in a lack of collaboration, nonstandard tools and disjointed processes. Agile attempted to
bridge these boundaries by breaking down some of the silos and moving individuals out
of these silos and into cross functional teams.
74
If done correctly, Agile can be very successful in breaking down these silos and making
organizations more productive and nimbler, but the truth is that very few organizations
have done Agile correctly. If we are honest, what we see is that we still have these same
silos inside the Agile teams. In other words, we have programmers who are responsible
for creating the application and quality assurance who is responsible for testing. All of
these individuals also still report back to the respective silo management where they
receive their directions.
What we see today in the DevOps movement is the trend to follow the same mistakes
that were made in Agile, the only difference is that these organizations in their infinite
wisdom have decided that it's a great opportunity to create a brand-new silo called the
DevOps department and which has a number of DevOps engineers. This idea for creating
a new siloed department is almost a guarantee that these organizations will never see the
full benefits that DevOps has to offer and most likely they will fall into the ninety percent
failure category that the DevOps movement is seeing.
75
The need for feedback loops
Feedback Loops in DevOps are how we put the Lean manufacturing principle of Kaizen
into place inside an organization. is arguably the most critical principle of lean
manufacturing. It should truly form the basis of your DevOps implementation. Without
continuous improvement, your progress will cease. As the name implies, Continuous
Improvement promotes constant, necessary change toward achievement of a desired state.
The changes can be big or small but must lend itself toward improvement. The process
truly is continual as there is always room for improvement.
Continuous Improvement should be a mind-set throughout your whole organization.

The way in which we implement the Kaizen principle in DevOps is through Feedback
Loops. One of the most misunderstood Feedback Loops in DevOps centers around
automated testing. Below we will hopefully clarify why the automated testing Feedback
Loop is so critical and why it is almost always never put into place inside most
organizations.
76
The majority of organizations implementing automated testing as part of their DevOps
implementation are still handing over the responsibility of automated testing to Quality
Assurance. These organizations continue down this path even though DevOps research
has shown that in almost all instances the automated testing effort will fail when
Development is not fully involved. So, what is the reason why the automated testing efforts
fail when Development is not fully involved, it's because we never create the needed
Feedback Loop between the automated tests and Development. Anyone who has ever been
involved with automated testing quickly understands the issue of trying to automate an
application that has not been designed to be easily automatable. This is where the
Feedback Loop comes in, unless Developers are involved with writing the automated test
cases, they will never feel the pain of trying to automate code that is not designed to be
automated. Another piece of this is the Data and Environment pieces of automated testing,
having known data sets and drift free environments is critical to the success of automated
testing, and generally only the Development team is capable of helping solve these pieces
of the puzzle.
Orca and Dolphin’s Project
Both companies have been tasked with creating a new online membership rewards web
site that their customers can use to track their rewards status. In many cases, both
companies may be fighting for the same customer. The goal is for each company to get
their new membership rewards web site launched to existing and potential customers as
fast as possible, while also infusing quality into the product and meeting the customer’s
needs. The fate of both companies is riding on this project.
77
Let’s Examine Orca’s Strategy in Depth
Iteration 1
Company Orca – Uses traditional Agile Scrum (2 week Sprints) with User
Stories and Acceptance Criteria
A. Company Orca creates four User Stories and Acceptance Criteria to initially develop
a new Membership Rewards Login Page and places them in the backlog.
B. The team determines during the sprint planning session that they can take in User
Story 1 and 2 in the first iteration.
C. The developers have a design session and start working on User Story 1 and 2
D. Due to difficulties with merging, building and environments, it isn’t until day 7 of
the iteration that QA is finally able to start their initial testing of User Story 1 and
2.
E. QA finds one issue and after scrambling, the team is able to get the fix completed
and back to QA to test
F. The Product Owner accepts User Story 1 and 2 during the Sprint Review
G. The code for User Story 1 and 2 remains in the team branch in the SCM
Breakdown of Iteration 1:
1.
2.
3.
4.
78
Learning Point(s):
DevOps Infrastructure
A critical factor in leveraging the benefits of Agile and DevOps is being able to implement
the tooling needed to fully utilize the DevOps feedback cycle in order to quickly respond
to business demand/feedback with high quality features/applications in a timely manner.
Unfortunately, an issue arises in many organizations that prevents the adoption, or the
leveraging of the tooling needed to create this DevOps cycle. The issue revolves around
outdated legacy databases/tooling/framework products.
All too often when an organization starts the process of moving to DevOps they discover
that they have some type of legacy product that stands in their way of fully adopting
DevOps infrastructure tooling, this product could be some commercial product or
homegrown product that the organizations has a critical reliance on. The product could
be used inside an organization for a myriad of things, business rules, a database, a load
balancer or number of other things. This product was most likely put into place during
the inception of an application(s) it supports and may or may not have been a smart
architectural decision at the time. The reality that the product is now limiting the abilities
to adopt desired DevOps infrastructure tooling is known by many involved in the DevOps
decisions, the problem is that there doesn’t seem to be an easy and cheap way to replace
it. This is actually a false premise that most organizations come up with.
79
The reason that this is a false premise is that when you actually look into the issue,
you realize that these organizations will commit a number of resources investigating all
the touchpoints for the product, looking at ways to possibly extract data from the product
and possible options to replace it, but there are a couple of other costly resource wastes
involved that these organizations may not be considering. Often if these organizations
would step back and apply critical scientific findings as Lean or Deming would suggest,
they would discover that the resource cost just to maintain this product is often staggering
and then there is also the cost involved in automating the infrastructure for the rest of
the framework, and also having to manually handle the building, integrating, versioning
(often likely there is no versioning or its strictly a manual versioning process) and
deploying of this legacy product.
If these organizations would take a fact-based look at the legacy product they have in
place, what they would most likely conclude that it's far more cost efficient to employ a
methodology such as the strangulation methodology to slowly replace this legacy product
with a newer product that affords them the opportunity to fully utilize and leverage the
available DevOps tooling.
There is one last conceptual issue we need to discuss on this topic that truly gets to the
heart of what DevOps is all about. In DevOps we are striving for turning the culture of the
organization as whole into a continuous improvement culture, but how is this possible if
we refuse to implement continuous improvement into our application framework/tooling?
The answer is that we can’t. To see this point all we really have to do is look at or read
about some of the organizations who have truly implemented the DevOps mindset and
what you discover is that they are not only always looking to improve the
framework/tooling backing their applications, but probably have already reversioned the
framework/tooling a number of times already. These organizations fully deploy the
concept of Evolutionary Architecture/Development as part of their continuous
improvement strategy.
80
DevOps SCM Strategy
While it may seem like having a Source Code Management strategy in this day in age
would be a given, the reality is that many organizations have given little to no thought
into their strategy. With the DevOps movement, having an SCM strategy has become even
more important. Instead of trying to technically explain how the various SCM tools work
and which is better, we will instead discuss some basics of why you need an SCM strategy
in DevOps.
Looking at the SCM strategy from a DevOps point of view, we need to realize that our
source code management tool is a natural meeting point for different roles and teams
inside the organization. It tends to be a little easier for more technical and software-
oriented roles and teams, it’s more difficult for other roles such as business analyst and
project managers. In order for the SCM strategy to be successful though, we must make
sure everyone is on board.
While it may seem like getting developers on board with the SCM strategy would be
fairly straight forward since they live and breathe source code, don’t be fooled into thinking
you won’t have to work to get them on board. The reality is that traditional methods such
81
as using long lived feature branches is ingrained in the way that developers have grown
accustomed to working and getting them to use new SCM tools such as GIT which is a
distributed version control system, and is designed to work locally with feature branches
will take work. A feature, or topic, branch is a branch that is used to keep track of ongoing
development regarding a particular feature, bug, and so on. This way, all changes in the
code regarding the feature can be handled together. Most organizations are finding that
GIT has generally become the go to SCM for DevOps, while the Git flow is a centralized
pattern and is reminiscent of workflows used with Subversion, CVS, and so on. The main
difference is that using Git has some technical and efficiency-related advantages.
One of the big differences in a DevOps SCM strategy occurs in Operations. The new
infrastructure code that is developed as part of the DevOps pipeline must be treated
exactly the same as application code. This means that in addition to the infrastructure
code being stored in an SCM, it must also be versioned and baselined just like application
code. The reason we version and baseline the code is to try and drive out the possibility
of Drift in our infrastructure. Drift is basically when we attempt to manage infrastructural
descriptors, which include network topology, versions of software that should be installed
on particular servers, and so on so that we do not get instance where our environments
are not the exact same. When our environments do not match up and we have introduced
significant Drift, then this becomes our main source of delivery costs and waste.
It would seem natural that as we move towards code based automated testing we would
store this could alongside our application code and version it, but the truth is very
different. A critical issue of storing automated testing code in an SCM has to do with an
issue we have previously discussed, Quality Assurance. Few Quality Assurance
individuals in most organizations have any practical experience working with SCM’s and
to make matters worse most organizations simply throw an SCM at Quality Assurance
with no formal training. The results are generally pretty obvious when Quality Assurance
starts working with an SCM, continuous merge conflicts and lots of frustration. If Quality
Assurance is going to be brought on board with using an SCM then they need to be
properly trained and mentored, this is even more pronounced when Specification by
Example tools are introduced into the equation.
One of the largest hurdles in an SCM strategy revolves around how to document all of
the steps required to perform various tasks. Organizations need to understand that this
part of the SCM strategy needs to emphasize tackling cultural issues and is not purely
technical one, but if it is not done properly it will have a negative impact on what happens
with the actual code.
82
The majority of organizations have attempted to employ solutions such as Wikis and
other team-based documentation tooling. The problem is that in most organizations this
is not a failsafe as often there will still be documentation floating around in documents on
shared drives and in emails. This is completely wrong from a DevOps point of view, instead
the documentation should be stored in the organization's central SCM where it is
accessible to all who need it.
Iteration 2
Company Orca
A. The team determines during the Sprint Planning session that they can take in User
Story 3 and 4 in the second iteration.
B. The developers have a design session and start working on User Story 3 and 4
C. While working on User Story 3, developer A1 who has been with the company many
years and has always worked on the feature area User Story 3 touches, realizes that
in order to finish User Story 3 he will have to make several changes to his existing
code.
D. Developer A1 works nights and the weekend to try and keep User Story 3 on track
E. Due to the delays in User Story 3, QA does not receive their first build to test until
the day before the Sprint ends.
F. QA finds an issue, but it is too late to get the fix in before the Sprint ends.
G. QA puts in a long night in order to test User Stories 1, 2 and 4.
H. During the Sprint Review, the Product Owner accepts User Story 4, but User Story
3 fails and will be carried over to the next Sprint.
I. Management commends developer A1 for his efforts to save the Sprint (Even though
it was his prior tight code coupling, poor design and lack of documentation that
caused the issue)
J. The code from the User Stories remains in the team branch in the SCM.
83
1.
2.
3.
4.
Learning Point(s):
84
Tightly Coupled Code Issues
One of the barriers inside most organizations that must be overcome is the issue of
applications and frameworks that were built with tightly coupled code. A number of
DevOps methods have been developed to try and overcome this issue, one of them being
the move towards micro services.
In Tightly coupled code, classes and objects are dependent on one another. In general,
tight coupling is usually bad because it reduces flexibility and reusability of code and it
makes changes much more difficult and impedes testability etc.
When we have a tightly coupled object, it becomes an object that needs to know quite
a bit about other objects and thus are usually highly dependent on each other's interfaces.
Changing one object in a tightly coupled application often requires changes to a number
of other objects. In a small application, we can easily identify the changes and there is
less chance to miss anything. But in large applications these inter-dependencies are not
85
always known by every programmer or there is a chance of overlooking changes. But each
set of loosely coupled objects are not dependent on each other.
In order to develop a DevOps pipeline and actually take advantage of its benefits, then
we truly need to move towards loosely coupled code. Loose coupling is a design goal that
seeks to reduce the inter-dependencies between components of a system with the goal of
reducing the risk that changes in one component will require changes in any other
component. Loose coupling is a much more generic concept intended to increase the
flexibility of a system, make it more maintainable, and makes the entire framework more
stable.
In DevOps, we find that a major killer in our move towards developing a pipeline that
allows us to quickly turn out high quality code is the affect tightly coupled code has on
Unit Testing. Make no mistake, if we do not have Unit Testing in place then it doesn’t
matter what tooling we create as part of our DevOps pipeline, we will still be dependent
on having manually test runs in order to get the code out. While having tightly coupled
code is not a complete death knell to Unit Testing, it does require organizations to
strategize their Unit Testing more closely and employ tooling such as SonarQube to
overcome this Technical Debt issue.
86
Overcoming the Cultural Roadblocks
It must be emphasized over and over in DevOps so that organizations do not make the
same mistakes that were made in the Agile methodology, unless you tackle the cultural
roadblocks initially then your chances of effectively implementing DevOps are slim to
none. Organizations must understand that in DevOps, tooling is not important, but
important. What we mean by this is that obviously to build out a DevOps pipeline we are
going to need tools, but to effectively use the tools they need to make the cultural changes
necessary to free their personnel to fully make use of them.
As an example, let’s take the implementation of a Continuous Integration tool such as

Jenkins. Jenkins is one of the more widely used CI tools on the market and most would
agree is very powerful and extremely versatile, yet most organizations who implement
Jenkins as part of their CI process are almost never able to leverage its abilities for the
simple reason that they have cultural issues preventing the adoption of Continuous
Integration.
What are some of these cultural issues preventing these organizations from leveraging
a tool like Jenkins in their CI process? Let’s take a look at some of the major cultural
roadblocks. The first is the complete lack of, or insufficient level of Unit testing. As
87
discussed earlier the reasons why Unit Testing has not been adopted by most
organizations has to do with cultural roadblocks to its adoption. If we were to look inside
most of these organizations, we would actually discover that most of the tooling needed to
perform Unit Testing is actually in place. It cannot be more clearly stated, without Unit
Testing in place effective Continuous Integration is dead in the water.
Another cultural roadblock to the effective use of a CI tool like Jenkins is the lack of a
Source Code Management strategy. As the name Continuous Integration implies,
developers should always be developing small increments of code that can continuously
be checked in, grabbed, built and tested, but the cultural truth is developers are so
ingrained in having long lived branches that a cultural shift must be made in order to get
them to move to checking in small increments.
Another cultural roadblock that has been discussed earlier is Quality Assurance and
Automated Testing. Again, it must be made very clear that there is no place in true
Continuous Integration for a manual test phase. It’s also critical to understand that any
Automated Testing that is included in the CI process outside of Unit Testing must be able
to run as part of the CI process and cannot be an independent process like some of the
commercial automated test tools require.
There are many more cultural roadblocks that must be overcome in most organizations,
but hopefully this example helps clear up the belief that simply implementing the various
DevOps pipeline tools without tackling the cultural roadblocks will never move an
organization forward with their DevOps implementation.
88
Flipping the Testing Pyramid
Perhaps the most difficult proposition to overcome in the move to Agile and DevOps is
the concept of flipping the testing pyramid or as it's also known inverting the testing
pyramid. In attending a number of Agile and DevOps conferences there is a common
takeaway, at Agile conferences there is normally a deafening silence from the various
presenters regarding the concept of flipping the testing pyramid and at DevOps
conferences it's very interesting to hear organization after organization get up and present
on the cultural hardships they have gone through on their DevOps journey and then
review the slick looking pipeline they built, but when they wrap up their presentation they
point out as sort of a side note that they actually only have maybe 30 or 40 percent of the
Automated Testing they need in place.
What you discover in the industry when you go speak with Agile organizations is that
they have a basic understanding that they need some Automated Testing and they have
mulled the idea or are doing some Unit Testing and they will broach the subject of
89
Functional Automation with tools like Selenium, but when they are presented with the
concept of flipping the testing pyramid they have a look of bewilderment on their faces.
On the other hand when you talk with organizations purporting to do DevOps you will
often find that they have a little or are starting to do Unit Testing and they also have
Functional Automation with tools like Selenium, but when you ask what they are doing
for security testing, front end performance testing, web consistency testing, automated
accessibility testing, load/stress testing and cross browser/os testing they either will state
that they have discussed some of these types of testing and some have no idea what they
are. One of the misconceptions you pick up on from talking with both the Agile and
DevOps organizations is that they believe when you discuss automated testing you are
discussing Functional Testing with tools like Selenium and that this Functional Testing
once implemented is going to magically solve all their problems.
So why have all these Agile and DevOps organizations been so unsuccessful in flipping
the testing pyramid? The reason is quite simple, it involves a cultural issue that most
organizations do not wish to bring up and many are simply unaware is an issue they need
to discuss. What is the issue? Its Quality Assurance. There are a few simple facts regarding
Automated Testing that simply cannot be overlooked anymore. First, it is just a reality
that true Automated Testing involves coding and the vast majority of Quality Assurance
professionals do not know how to code and learning how to code requires more than one
automated testing course. Secondly, Automated Testing to be used effectively needs to be
incorporated into the Continuous Integration process, which requires familiarity with the
tools used in the Continuous Integration process. To demonstrate this simply ask most
Quality Assurance professionals to explain what the Maven lifecycle is and then watch the
blank look they give you. Thirdly, bring up the subject of how to do automated security
testing or better yet how to do automated front end performance testing to Quality
Assurance and they most likely will have little idea of what you are talking about or the
tooling necessary to perform it. The last reason is that Agile preaches that the whole team
is responsible for quality and that Quality Assurance is no longer the quality gatekeeper,
but if you go into these organizations they still treat Quality Assurance as the owner of all
testing and the quality gatekeeper.
What all of this means is that until organizations are willing to have an open and honest
conversation about the role and responsibilities of Quality Assurance in Agile and DevOps,
few organizations will be able to effectively flip the testing pyramid.
90
Components of Change
Wagner, John A, and John R. Hollenbeck. Management of Organizational Behavior. Prentice-Hall,

1995.
Reading a number of DevOps books and listening to presenters at various DevOps

conferences you consistently hear about the Blame Game and the need to bring
development and operations together by implementing change through incentives so that
their objectives are inline. Yet, when you go into organization after organization that is
purporting to be moving to DevOps and you ask about a plan for this change what you
find is that management has not even broached this subject and often will state they see
no need to change the incentive structure.
Changing the incentive structure is quite possibly the most important role executive
management may play in nurturing the growing DevOps movement within an
organization, but in order for this incentive change to have any chance of success all of
the other components of change must also be in place. In order to explain why it’s so
critical for executive management to change the incentive structure we will discuss the
required components of change in our diagram above and how they relate to DevOps.
Our first component is “Vision”. As our diagram above shows, when Vision is missing
inside an organization attempting to move towards DevOps, it results in confusion. What
we find is that when the innovators leading the DevOps movement do not properly
91
evangelize the vision behind why it so critical to move to DevOps, this will lead to confusion
in the organizations, which in turn opens up opportunities for the laggards to sabotage
the movement. There are numerous ways of evangelizing the vision inside an organization,
but here are a couple of keys to being successful. The management must not only
understand the vision but must be able to effectively communicate the supporting facts
guiding it. This means they need to understand and be able to backup why the vision is
needed through scientific facts as Deming has stated. Management must also lead by
example, meaning if the vision for moving to DevOps states it’s to have the highest quality
product, but management only pays this lip service the first time an incident occurs then
the rest of the organizations will not have the necessary belief in the sincerity of the vision.
The next component of change is skills and when the required skill set is missing in
DevOps we often see as a result “anxiety”. A few examples where you see anxiety result
from missing skill sets in DevOps are Quality Assurance, developers displaying hero cult
tendencies and managers who don’t have the necessary understanding of the foundations
of DevOps. The move to DevOps brings about the need for Automated Testing, which
requires coding skills, familiarity with new tooling and a new mindset regarding how to
test. These new skills, tools and testing mindset fly in the face of traditional Quality
Assurance and are most often meet with anxiety by traditional QA professionals who worry
about their job security and ability to learn these new skills, tools and techniques. The
developers who exhibit hero cult tendencies are normally anxious about the organizations
discovering their true skill level or their lack of architectural knowledge. Managers who
don’t have an understanding of the foundations of DevOps will often unfortunately turn
to micro managing with the hope of being able to control the change that is occurring
around them that they simply do not understand.
The next change component is “incentives”. When attempting to implement change but
ignoring the need to adjust the incentive structure to support the desired change results
in resistance. Management needs to learn that changing incentives actually can have
positive effect on behavior and performance and attract new resources and capabilities,
which can lead to culture change. Organizations need to be clear that they can implement
any number of DevOps tools intended to initiate change and improve operations, but
incentivizing people the wrong way leads to inefficiency and unproductive business
outcomes in all instances. Management must also be aware that Incentive misalignment
reinforces silos, which is in direct conflict with the principles of both Agile and DevOps.
Two important points when determining how to change incentives. first, Reward publicly,
not privately so everyone knows what work is appreciated and why. A regular public
reminder works better than a private one. Secondly, Reward behaviors, not only outcomes.
92
Outcomes can often be achieved through shortcuts while behavior is about hard work and
effort. remember, DevOps is about driving out fear and encouraging the idea of fail fast
and learn from it.
The next component of change that needs to be addressed is resources and the
frustration that is caused when there is a lack of resources or perceived lack of resources.
All too often the management at all levels in an organization undergoing a DevOps
transformation will experience the real or perceived belief that they lack the necessary
resources to complete the DevOps transformation they have undertaken. This frustration
over resources can lead to some damaging decision making that negatively affects the
DevOps transformation. Management in their belief that they lack the necessary resources
may decide that they need to outsource or bring in outside resources to handle parts of
the DevOps transformation they don’t feel they have the necessary resources to
accomplish the needed work, this is a mistake that far too many organizations make.
Simply put, DevOps needs to be accomplished from within and trying to farm out key
components of your critical infrastructure will likely lead to failure. All too often
management is their worst enemy when they perceive that they lack the necessary
resources to accomplish something and, it’s their own misjudgment that is causing this
perceived lack of resources. An example seen in many organizations is the implementation
of Unit Testing where management decides that Unit Testing is needed and comes up with
an implementation plan and starts to implement it, but suddenly an unexpected issue
arises, and management decides that this issue must be addressed immediately and the
already planned work must also get done, so the only way to do this is to put Unit Testing
on hold. The truth is Unit Testing never gains momentum again and withers and dies.
Management must instead stop trying to put band aids on everything and instead pull off
the band aid and suffer the pain and do the right thing and solve the underlying cause of
the wounds in the first place.
The last change component is having an action plan. During a DevOps transformation
lacking a well evangelized and transparent plan causes enough false starts that almost
surely will allow the laggards to sow enough doubt into the minds of the followers and
management to effectively kill any hope of DevOps. It’s often very frustrating to discuss
the need to evangelize and make transparent the DevOps plan and its progress and
failures, because all too often organizations and individuals feel that this is almost sort
some kind of DevOps fluff and isn’t that critical. The problem is that when you try to point
out that the vast majority of organizations will fail in their quest for DevOps and that when
you look at the few organizations that have succeeded they have a common trait and that
is that they are constantly evangelizing the DevOps progress to everyone in the
93
organization through a myriad of ways. If one were to visit many of these organizations
they would notice the tv screens everywhere broadcasting the DevOps progress, metrics
and results throughout the organization's complex. Evangelizing the DevOps action plan
is not fluff, but a critical factor in the success or failure of DevOps.
Iteration 3
Company Orca
A. A total of 14 User Stories are created in order to develop the Membership Rewards
Home Page and they are put in the backlog.
B. After a Pointing session, the team takes in 3 User Stories and User Story 3 that was
carried over from the previous iteration.
C. The developers have a design session to flush out the details
D. Continuing problems with merging and building environments again delays the final
testing of User Story 3 that was carried over.
E. On the 4th day of the Sprint, User Story 3 is finally able to be tested.
F. An urgent hotfix is needed, and two developers are temporarily assigned to handle
the hotfix
G. The remaining developers start turning over code to be tested to QA on day 6 of the
Sprint, but have to work evenings in order to make up for the two developers who
were pulled to handle the hotfix
H. The QA group is able to test all four User Stories from the Login Page and the three
new User Stories.
I. During the Sprint Review, the Product Owner accepts all of the stories
J. The code from the Stories remains in the team branch in the SCM.
94
1.
2.
3.
4.
Learning Point(s):
95
Using Kaizen to Fight Waste
Bittner Follow, Kurt, and Andrew Phillips. “Building a Compelling Business Case for Continuous Delivery.” LinkedIn
SlideShare, 7 Oct. 2015, www.slideshare.net/xebialabs/building-a-compelling-business-case-for-continuous-
delivery?qid=166bee73-4beb-4a13-98f5-9b567c493419&v=&b=&from_search=2.
If most organizations were to take a very close view of their value streams, what they
would discover is that they have waste strewn throughout the entire value stream. The
only way in which these organizations are ever going to effectively tackle this waste is by
changing to a Kaizen culture. Again, if we are to truly understand DevOps, then we must
fully understand one of the key principles that DevOps is built on and that is Kaizen.
96
Kaizen is a Lean tool/concept for the elimination of waste through continuous
improvement. Let's take a close look now at all of the areas of software development and
the typical waste in the entire value stream, the causes and how to eliminate the waste.
One area that very few organizations have realized has excessive waste is the inception
phase. The inception phase of the value stream generally incorporates customer need
comprehension, solution generation and solution proposal (Agile artifact creation). In this
phase, what research has shown and is not widely publicized is that up to two thirds of
this work that is developed turns into waste. Why would almost two thirds of this work
we are creating for our Agile teams be waste? There are a few key reasons for this, one is
that feature creep is a much more prevalent problem in the industry then the vast majority
of organizations realize. Part of this problem is that almost no one has put any type of
metrics in place to measure how effective product management is in meeting customer
needs over time through the Agile artifacts they create. The second reason for this waste
also ties into the first issue of feature creep and DevOps attempts to fix this through
feedback loops. DevOps calls for short iterations with small increments of code moving
through the system all the way to the final customer with feedback loops and monitoring
in place, plus the possibility of A/B testing (which will be covered later in the book) so
that Agile teams can get quick feedback on what they have created in order to quickly
make adjustments based on actual customer feedback. This is in stark contrast to
traditional development where teams work for weeks, months even years to complete all
of the Agile artifacts and then finally release to customers only to find out they have not
meet customer expectations or created something the customer didn’t even want.
The next area of waste in the value streams of many organizations is the continued use
of manual builds. One often unforeseen case of waste when a manual build process is
used can be seen in the resources wasted having to test every build just to confirm that
it's even testable. Many organizations using a manual build process will most often assign
one or two manual testers to run through and confirm the most basic functionality is
working so that the build can be used in testing. It is not uncommon for these manual
testers to spend upwards of four hours each week, doing this week after week just to
confirm builds are testable, if these same organizations would automate the builds and
just add simple automated smoke suites, they could realize a tremendous ROI almost
immediately on their automated build efforts. This waste also extends into development
where we often see developer resources wasted having to manually go through the manual
steps to create a build and also Operations resources waste if they are tasked with
deploying the build. Add all these resources up every time a build is created and deployed,
97
and the manual testers discover that something got left out of the build and the everyone
has to go through the same process again.
The next area of waste in most organizations value streams is testing and as is argued
in this book possibly the biggest constraint in the value stream. A point that first needs
to be made here and understood by all and shown in research over and over is that manual
testing will never improve quality, instead it just highlights the lack of quality in the
products being created inside organizations. The problems with manual testing are
numerous in the industry, the process is incredibly slow, it is error prone, many of the
individuals in the field are not formally trained and manual testing reinforces the creation
of silos. When looking at manual testing one also has to take into account that this type
of testing is incredibly expensive while at the same time not providing the necessary
coverage that is truly required. If most organizations take a fact-based look at manual
testing what they would discover is that the manual testing only covers a small percentage
of the overall testing that should really be required to get a feature or application out.
Our last area of waste in the value stream is one that unfortunately can have a ripple
effect all the way from the build process to production support where it is responsible for
anywhere between twenty and eighty percent of production incidents. The cause of this
waste is what is termed “drift”. Drift is the general tendency of software and hardware
configurations to drift, or become inconsistent, with the template version of the system
due to manual ad hoc changes (like hotfixes) that are not introduced back into the
template.
98
Moving from a Ticket Taking Culture
An often-heard refrain you hear in DevOps books and presentation is the idea of
software development moving from seeing itself as sort of a ticket taking function that
supports the core business, to instead seeing itself as a central part of the business core.
As some have pointed out when using Lean Manufacturing as an example, software
development has become the factory floor of modern businesses and without it the vast
majority of businesses wouldn’t be able to function.
To understand this, all most organizations have to do is examine where their new
business comes from, many organizations already do this but still fail to understand the
importance of software development as a core component of the business. Unfortunately,
when most organizations attempt to move to DevOps, they discover that the business side
of the organization often fails to not only recognize the importance of software development
on the business bottom line, but they continue to view and treat software development as
support role for their needs.
99
In order for DevOps to truly be successful, the reality is that the leaders of the DevOps
movement must be able to present to executive management a true picture of why software
development must be treated as a core piece of the business. An often-seen mistake is for
the DevOps leaders to do a presentation to management and believe they can rely on
showing management the benefits to the business of moving to DevOps. The DevOps
leaders who follow this tactic will discover that the majority of time all this leads to is the
business wanting the benefits shown, but not understanding and providing the needed
support (financially, strategically, culturally, structurally and resources) to make DevOps
a reality. To be successful, the DevOps leaders need to provide the big picture of the role
software development plays in generating business and then explain with the support of
management and the rest of the business how they can expand the business, but only if
the DevOps movement has the necessary support.
Agile Best Practices
100
One of the realities of DevOps is that we really can’t be successful with DevOps unless
we also fix what is broken in the Agile methodology being used at most organizations. One
notion we need to be fully cognizant of is that much of Agile's popularity in Software
Development is due to the fact that it’s a fairly easy sell to the organization's business side
because it promises solutions to many recurring IT concerns. These recurring IT concerns
include projects that constantly run over budget and time, lack of team effectiveness, lack
of true collaboration, poor product quality and dissatisfied customers.
A simple internet search for common Agile problems will show list after list of top 5, top
10 and top 25 common Agile problems, but for our sake we will concentrate on some
general high-level problems that have caused the Agile movement to jump the track. The
most common problem with Agile is the exact same problem that affects DevOps and that
is complete lack of understanding of the principles of Agile and foundations in which it is
based on by management. Organization after organization has attempted to move to the
Agile methodology for the simple reason that someone in management has heard the Agile
buzz word and has convinced others that this is the direction that they need to go in.
Again, we see management in these organizations disregarding their duty to do due
diligence in exploring and investigation Agile and what all is involved and what is needed
to make it successful. It’s fairly common practice for most of these organizations to bring
in an Agile consultant to help them implement their chosen Agile methodology and to
initially start the Agile journey on the right foot, but unfortunately, the minute the
consultant steps out the door the organization starts ejecting the hard and unpopular
pieces of Agile that are balked at by teams, groups or individuals. The results are that
within a short amount of time what is left in place resembles Agile only in name.
Another common problem even though it’s almost hard to believe that many
organizations don't understand this common-sense problem, the problem of not having a
product owner. Of all the things that can cause an Agile software project to fail, not having
a person that is ultimately the decision maker for the product being developed is the
quickest way to ensure its demise. If you want your project to succeed, then you absolutely
need someone who can set its direction and make decisions about the product being
developed.
Going back to the problem of not fully understanding the principles of Agile and DevOps
is the common problem of not understanding what true iterating is. In most organizations
the vast majority of individuals believe that iterating means breaking your project into 2-
week sprints or iterations. While we are not saying that doing this can’t facilitate iterative
101
development, it doesn't guarantee you are actually iterating. The principle of iterating
preaches that features should not be built in a single sprint or iteration. What we actually
should be doing if we are truly iterating is to see, features that evolve and grow over time.
It should be hard to picture any feature that can be developed in a single sprint or iteration
and then be done. instead features should start out small and develop and evolve over
time as the team receives feedback and customers or the product owner try it out.
Looking at Agile in many organizations we also see the problem of not breaking work
(stories/cards) down small enough. In Agile we need to break work down into small, easily
digestible pieces. You often see the group doing backlog grooming not considering the
work items size/complexity enough before making them available to the team. These
backlogs are not broken down small enough and often are generally vague in what has to
be accomplished. These backlogs are notoriously difficult to estimate and waste the
various teams time when trying to understand them. It is critical that backlogs are broken
down into smaller actionable backlogs before being given to an Agile development team or
a large amount of time will be wasted.
The next common problem we see in Agile by organizations is simply not setting done
criteria. Agile requires that the product owner should be defining some level of acceptance
testing. It’s great practice to set this acceptance testing through the use of Specification
by Example. In addition, these tests really should be fully Automated Tests. What matters
most is that some done criteria are defined by which the team can evaluate whether they
have achieved their goal or not.
The last problem we will discuss is something that Agile was actually designed to help
solve inside organizations practicing Agile and that is letting teams be teams and in turn
break down silos. A key to having healthy and motivated agile teams, is to allow them to
be mostly autonomous and out from control of traditional silos. When silos are still
controlling the individuals inside agile teams and they don't have the power to decide their
own fate, team members will look out for themselves and have loyalties to their silo and
not their fellow team members.
102
Iteration 4
Company Orca
A. After a Pointing session, the team takes in 4 User Stories for this iteration.
B. The developers have a design session to flush out the details.
C. The developers start turning over code to be tested to QA on day 5 of the Sprint
D. The QA group is able to test all four User Stories from the Membership Rewards
Home Page, and the previous seven User Stories.
E. During the Sprint Review the Product Owner asks for a specific scenario to be run
for User Story 10, the scenario does not work as the product owner expected. User
Story 10 does not pass. The team and product owner trade blame for why the
scenario was not covered during testing. The scenario was not highlighted in the
acceptance criteria. But the product owner believed the team should have known to
test this scenario.
F. The final result is that the Sprint fails, the Product Owner accepts 3 User Stories
and User Story 10 will be carried over to the next sprint.
G. The code remains in the team branch in the SCM
1.
2.
3.
4.
103
Learning Point(s):
Specification by Example
Specification by example refers to the use of relatively concrete examples to illustrate

how a system should work, as opposed to more formally written specifications expressed
in very general terms. Specification by Example is also often interchangeably referred to
as Behavior Driven Development or Acceptance Driven Development and Acceptance Test-
Driven Development.
There are some very compelling reasons for moving to Specification by Example which
we will discuss first and then we will wrap up by discussing the critical cultural change
that it can enable. One major benefit of Specification by Example is the transparency it
can bring to the testing component of DevOps. By incorporating Specification by Example,
104
we allow the product owner to be able to know precisely which features are being delivered
in a particular release, and also know what works and what doesn’t via viewing a simple
English like test. Business Analyst are able to see exactly how stories have been
implemented. The Operations team will want to know exactly which features are coming
down the pipeline to be delivered into production.
Another benefit is that Specification by Example is that it serves as a vehicle to foster

communication between all stakeholders by serving as a common language. The use of
Specification by Example forces the use of common verbiage in the organization in order
to allow frequent conversations between domain experts and the delivery team so that
organizations can lay a foundation for the entire development process.
Specification by Example allows the examples you write to become living

documentation, this is accomplished through the automation of the examples, which keep
specifications in check when run consistently in the DevOps pipeline. The frequent
validation via the tests in the pipeline allows organizations to know that the examples,
and conversations are up to date, and when you trust your tests, you can use them as
documentation for the entire system. A side benefit is that you can track every example
back to the tests that make up that example, which allows teams to know exactly which
tests need to be updated whenever an example is changed, replaced or deleted. By
validating frequently, we guarantee that the documentation must change every time the
product changes.
Possibly the most important component of Specification by Example and probably the
least discussed is its ability to become an agent for cultural change inside an organization
moving to either Agile or DevOps. By using Specification by Example, we can break the
traditional stronghold the Quality Assurance silo has had on testing. When we start using
examples inside our Agile teams and get the developers and business side actively
involved, what we discover is that this can break down the resistance developers often
have to taking over Automated Testing because with the examples in place the developers
often no longer will push back with the concern that they will have to spend time
attempting to figure out what needs to be tested, the exact tests and expected outcomes
are now laid out for them. In addition, we also break the traditional way in which test
cases have been written by Quality Assurance, we focus on the critical behaviors that are
called out by the business side through the example creation process and we no longer
attempt the blanket approach to testing.
105
Define CI and CD
In many IT books you can read suggestions for what tools should make up your
Continuous Integration (CI) or Continuous Delivery/Deployment (CD), but what is often
overlooked by organizations attempting DevOps is to define what constitutes CI and CD.
Before getting started on building out the CI and CD portions of the delivery pipeline, we
highly suggest organizations first define what constitutes CI and CD and then allow teams
to experiment with tools to build out the CI and CD portions of the pipeline.
By defining CI and CD you give teams a standard in terms of what the CI and CD
processes must entail in order for them to meet the organizations definition. Defining CI
and CD inside the organization does not mean that the organization can just make up any
definition that suites them, the definitions must at least meet the industry minimum
definitions. As an example, CI would most likely need to at least include a code
management tool that allowed teams to work off the mainline, an appropriate level of Unit
Testing, other Automated Testing that the organization determined should be included,
code quality gates and versioning tooling at a minimum.
Once the CI and CD have been defined, then teams should be allowed to experiment
with tooling that works for them and meets the defined steps in the CI and CD processes.
106
Shift Left and Continuous Testing
Shift Left Testing
The practice of Shifting Testing Left is intended to find and prevent defects early in the
software delivery process. This idea borrows from the teachings of Dr. Deming in his 14
points that we need to prevent defects instead of detecting them. The idea is to improve
quality by moving tasks to the left as early in the lifecycle as possible. Shift Left testing
means testing earlier in the software development process.
In the traditional software development V model, requirements are kept on the left side
of the plan, and the delivery and testing requirements are on the right side. The problem
is that these practices can’t handle changing expectations and requirements, and they
lead to negative outcomes for the business such as increased costs, increased time to
107
market, and unexpected errors. An excellent breakdown of the “4 types of shifting left
testing” are covered by Donald Firesmith from the Carnegie Mellon Software Engineering
Institute.
Continuous Testing
The term Continuous Testing refers to the process of executing automated tests
throughout the delivery pipeline, thereby giving teams the earliest possible feedback
(Feedback Loops) about the quality of the code, and timely information about the risks
associated with deploying that code to production.
Continuous Testing also takes into consideration all of the measures that an
organization takes to build testing into the development and delivery process, from
specification to design, to coding to integration to production.
Continuous Testing is often thought to be the same as test automation, but a more
expansive definition includes activities that cannot be automated but can be “shifted left,”
such as the integration of testability into the design process or the earliest possible
acquisition of user or customer feedback.
108
Let’s Take a Break and Play a Game
The Envelope Game
One of the key principles of both Agile and DevOps is the idea of Agile teams working
in short iterations to release code incrementally and iteratively in order to streamline the
development process and to enable us to release software automatically. We will
demonstrate a simple envelope game below that readers may use inside their organization
to demonstrate the advantage of Agile teams releasing code in small increments versus
the traditional big bang release strategy.
To illustrate the advantages of releasing software incrementally and iteratively versus

the traditional big bang method we will step through the simple envelope game below.
The envelope game used here is one that really shows the advantages of incremental
and iterative work versus the traditional big bang and is not a "manufacturing" example,
so it can be used with about any audience and only takes about 20 minutes:
You'll need about 10-12 participants for this game:
For the game you will need 2 "teams" of 5 people each to volunteer and play along. In
addition, you will also need one individual to play the role of the customer. If you have
enough people, assign a "timer" to each team to record the step-wise cycle times.
109
Have each team sit or be arranged in a row so they can hand items down the line. Each
'team' member will need a pen and a block of sticky notes (Post-Its need to be in two
different colors). You will also need a total of ten envelopes. The Customer should sit at a
separate location.
The target or goal is to produce 5 complete orders, error free, where an order is an
envelope stuffed with a sticky note with each of the 5 weekdays written on it (Monday,
Tuesday, Wednesday, Thursday, Friday)
Team A's process is for person 1 to write 'Monday' on each of 5 sticky notes. Only when
all 5 have 'Monday' written on them can person 1 pass the orders to person 2. Person 2
writes 'Tuesday' on each of the 5 notes and can only pass to person 3 when all 5 have
been done. This continues across all 5 people on the team. Once the 5th person completes
their task (writing 'Friday' on all 5) they will put one sticky note in each envelope and hand
the "orders" to the customer and stop.
Team B's process is for person 1 to write " Monday" on the first note, then pass it to
person 2. Person 2 writes "Tuesday" on the note and then passes it to person 3... and so
on. After person 1 has passed the first note to person 2, they begin work on the second
note and pass that along once their task is completed. Person 5 will write ‘Friday’ on the
first sticky note and put it into the envelope and hand it to the customer. This continues
until Team B's person 5 has written 'Friday' on the 5th note and put it into an envelope
and handed it to the customer.
In order to throw a little wrinkle into the game and truly show the advantages of working
in an incremental and iterative fashion and to also highlight a DevOps feedback loop and
demonstrate the waste involved in the traditional big bang method we will have the
customer tell person 5 for team B that they wish to have the weekdays written on the
second color sticky note instead. Person 5 will go back to Team B and inform them.
Remember team A will not know about the customer’s preference for the different color
sticky note until they turn in all of the orders at once. When they turn in all of their orders,
the customer will then inform them of their preference for the different color sticky note
and they will have to start over.
You then, along with the times and balance of participants graph the step by step cycle
times and the overall cycle, or completion time so they can all clearly see the difference
between the traditional big bang processes and the incremental and iterative flow.
110
You can then follow this exercise up with some facilitated discussion to get the
participants to talk through how the processes inside the organization currently works
and how they could improve their work by using the incremental and iterative process.
Iteration 5
Company Orca
A. After a Pointing session, the team takes in 4 User Stories for this iteration and User
Story 10 from the previous Sprint.
B. The developers have a design session to flush out the details. The developers start
turn over code to QA on day 3 for the changes to User Story 10, the User Story
passes testing.
C. The developers start delivering code for the other 4 User Stories on day 5 of the
Sprint.
D. The QA group is able to test all four User Stories from the Membership Rewards
Home Page in the current Sprint and the previous eleven User Stories.
E. The Product Owner accepts all User Stories from this iteration.
F. The code remains in the team branch in the SCM
1.
2.
3.
4.
111
Learning Point(s):
Hidden Constraints
One of the starting points with any organization looking to start the DevOps journey or
even just looking for process improvement is to create a current and future state Value
Stream Map or apply the Theory of Constraints. The concept of Value Stream Mapping
has been around for many years (possibly as early as 1918) and is usually associated with
Lean. Value Stream Mapping can be used in conjunction with the Theory of Constraints
or they may both be applied separately. Value Stream Mapping helps show how both
materials and information flow as a product or service through the process value stream,
helping teams visualize where improvements might be made in both flows. The Theory of
Constraints is used to concentrate on reducing the throughput time. By optimally
112
exploiting the bottlenecks or constraints, the efficiency of the process as a whole is
improved.
There are a couple of issues practitioners should be aware of when applying Value
Stream Mapping/Theory of Constraints to Software Development. First, Value Stream
Mapping and the Theory of Constraints have traditionally been applied to manufacturing
where outputs are tangible and easier to measure. Second, in manufacturing we would
look out for “Hidden Constraints” which are constraints in the business process which
turn out to be based on wrong assumptions. An example of such a ‘hidden’ bottleneck
could be introducing replenish-to-consume everywhere in the supply chain.
When creating a Value Stream Map in Software development or applying the Theory of
Constraints we may have a Hidden Constraint/process improvement that we are not fully
seeing. When using the Theory of Constraints, we will probably identify “testing” in our
process and then identify it as one of our constraints and with Value Stream Mapping we
would identify testing as a process improvement. We may identify other areas in the
process as bigger constraints/areas for process improvement then testing, this may be
because testing is a like an iceberg. What we mean by this is that testing may be much
bigger constraint/improvement area then you are actually seeing. Let’s take for example
a traditional organization who does manual testing and has does a traditional test phase.
If we take a very close look at this test phase and are honest we will probably find that the
testers are not doing nearly the depth or width of testing that is really required. What do
we mean by this? Well, check and see if they are doing consistency testing across all the
browser/os combinations you support, are they doing security penetration testing, do they
do front end performance testing or are they doing testing beyond simple happy path
testing, do they do web/mobile functional testing across all browser/os combinations
supported. (We won’t even get into Unit testing)
This is not the fault of testers, it’s just the reality of attempting to cover testing in a
manual way with limited resources and time. So, when you do your Value Stream Mapping
or attempt to apply the Theory of Constraints, dig under the covers a little and keep an
eye out for hidden constraints/process improvements.
113
Iteration 6
Company Orca
A. After a Pointing session, the team takes in 3 User Stories for this iteration.
B. The developers have a design session to flush out the details. Due to difficulties with
merging, building and environments, it isn’t until day 7 of the iteration that QA is
finally able to start their initial testing.
C. The QA group is able to test all four User Stories from the Login Page, the previous
eight User Stories and the 3 User Stories for this Sprint.
D. The Product Owner accepts all 3 User Stories for this iteration.
E. The code remains in the team branch in the SCM
1.
2.
3.
4.
114
Learning Point:
Conway’s Law
Conway, Mel. Conway's Law, melconway.com/Home/Conways_Law.html.
Conway’s Law is seen as more of an observed phenomenon. This phenomenon is that

the products software teams tend to create, often reflected their organizational
communication structure. The reason why it’s important to understand Conway’s Law
and be aware of how it could affect your DevOps movement inside your organization is
because it goes straight to the point of how organizations might use Conway’s Law to drive
their possible DevOps organizational change. The key question Conway’s Law brings up,
is there a need to change the organizational structure in order to build better software?
Out of Conway’s Law has come the term “Inverse Conway Maneuver from Jonny Leroy
and Matt Simons of ThoughtWorks. The Inverse Conway Maneuver states that you may
want to begin by breaking down silos that constrain the team’s ability to collaborate
effectively. Another way to think about this is that Conway’s Law is about building
dysfunctional software.
A noted problem with Conway’s Law is that it does not provide any diagnostic tools to
help management teams determine whether or not their organizations are properly
115
structured, it also fails to identify when a reorganization of a company might make sense.
One question it does raise in DevOps, does the current organizational structure lend itself
to building a quality product the customer actually desires? Conway’s Law should also
encourage organizations to contemplate investigating the scientifically based field of
Organizational Development before undertaking any organization change.
Leroy, Jonny, and Matt Simons. “Dealing with Creaky Legacy Platforms.” Making Things Better, 3 Feb.
2011, jonnyleroy.com/2011/02/03/dealing-with-creaky-legacy-platforms/.
Iteration 7
Company Orca
A. The Sprint team now starts the tedious process of merging the feature branch back
into the Trunk.
B. The QA team adds additional resources for the Regression/Integration/System
testing phase.
C. The development team is finally able to merge the code and deploy to a staging
environment after 5 days.
D. The QA group starts testing on the staging environment and after a couple of hours
of testing discovers that portions of one of the features did not get included in the
merge and deployment
E. The development team investigates the missing code and is able to determine the
issue and include the code in a new deployment to staging.
F. The QA group once again start their testing.
G. The Sprint team determines that they would like to perform some Performance
testing on the new Membership Rewards features, but is told they need to discuss
this with a QA member who is on another team
H. A release handoff meeting is scheduled to determine when the new Membership
Rewards features will be released, coordinate with IT/OP’s and make sure the
change management process has all of the required sign offs.
116
1.
2.
3.
4.
Learning Point(s):
117
Versioning
Version management is a required aspect for any DevOps deployment pipeline. For our
discussion purposes, we will state that any object that can flow through the delivery
pipeline should be versioned. We can define an object as everything that is deployed
through the deployment pipeline, examples would include software application code,
software configuration files, middleware, middleware configuration files and hardware
configuration files. We also need to understand that any other work that is created in
order to manage the deployment pipeline should be considered objects, examples include
themes, epics, features, stories, requirements, specification by example, test cases and
any required documentation.
So, what exactly is versioning? Versioning is a way to categorize the unique states of
software as it is developed and released. We should also understand that there is no
universal standard for how to assign versioning, some general logic is using a major
release number, minor release number, maintenance release number and build number.
There are a number of common sense reasons why we should version, they include:
Prevent overwriting of changes by others

118
Allow for root cause analysis of bugs
Determine object version in various environments
Determine who made a change to an object and when it was done
Integration of objects
Lastly, we should identify what objects should be versioned.
Agile Artifacts
Software
Data models
Baselines
BuildID
Specification by Example/Test cases
Test data
Test runs
Configuration files
Deployment Scripts
Release Notes
Data files
Protocols
Monitor Configuration files
119
Change Management
The concept of Change Management should not be looked at as something that is going
away in DevOps, instead it should be viewed more, as a way of improving the verification
process, saving resources, using the pipeline as the change management tool and
streamlining the process.
DevOps breaks down Change Management in three ways:
Risk reduction – DevOps reduces risk via the introduction of robust Automated Testing of
both application and infrastructure code, the rapid detection of problems through greatly
enhanced monitoring and feedback loops and an enhanced rollback ability.
Constant builds – A key principle underlying DevOps is that we take things that are
painful such as builds, versioning and deployments and do them constantly in order to
learn how to do it better and take the pain away.
The process is the documentation – DevOps is a highly automated, end-to-end process

from inception through, testing, deployment and support and as part of the process we
build in checks and balances through testing, feedback loops and monitoring that are
required for compliance in the change management process. In DevOps, instead of having
120
a highly manual documentation process, the automated workflow becomes the
documentation process.
DevOps over time will change the role of Change Management. Over time the role of IT
Operations will change from a role that acts sort of as a gatekeeper that inspects every
single code snippet that flows through the pipeline, and instead will turns its attention to
focus on the pipeline process itself. By focusing on the pipeline process, itself, they will
now work with teams to help design and build a pipeline in such a way that what comes
out the other end meets the quality, risk and compliance criteria for Change Management.
In addition, the IT Operations role should move the focus of their attention to ensuring
that the pipeline has the necessary checks built in to ensure that teams aren’t skipping
steps and or gaming the process. You see this in some organizations where the pipeline
process has built in testing such as Mutation Testing to test the Unit Tests being written
by teams are actually testing components as expected.
Provisioning
121
In order to deploy applications or setup automation in DevOps, we need a way to be
able to setup the infrastructure for our environments. These environments include Dev,
Test, Staging, Production, etc. This is where Provisioning comes in.
All of the environments in our delivery pipeline should be perfect reproductions of

production in order for us to catch all the bugs that might occur in production and easily
reproduce them in any environment. This means we must ensure our infrastructure is
provisioned exactly the same way each time to avoid problems due to configuration errors.
Provisioning infrastructure manually is simply not viable due to it being such a time-
consuming process and it is also impossible to guarantee zero errors this way. In addition,
manually provisioned environments tend to ‘drift‘ due to ad-hoc changes over time. The
wider the gap grows, the more you will run into unexpected issues and failed deployments.
By automating the provisioning of infrastructure, we eliminate configuration drift,

ensure predictability in what was provisioned, and thus make our software delivery
process much more reliable.
122
Iteration 8
Company Orca
A. A date and time are selected for the release of the new Membership Rewards
application.
B. On the evening of the release, the Sprint team that developed the new Membership
Rewards application is required to monitor and test the new deployment.
C. No issues are discovered during the release and subsequent testing.
D. Several days after the release, the IT support team starts receiving complaints from
customers about the responsiveness of the Membership Rewards application
E. The issue is turned over to the Op’s team to investigate, support from development
is requested. Almost immediately blame between the Op’s and Development teams
starts.
F. After several days of arguing and researching the performance issue, it is discovered
that a web service support plugin that worked correctly in the development, QA and
Staging environments does not work in Production due to differences in their setup.
G. A work around for the issue is pushed to Production and appears to have resolved
the performance issue.
H. The Sprint team that developed the new Membership Rewards application now goes
into a series of iterations fixing bugs reported from customers and starts altering
the features as requested by the customers.
1.
2.
3.
4.
123
Learning Point(s):
Blame Game
If you have worked in software development for any length of time, then you have seen
the following scenario, a new software release has just gone live, and shortly after that the
clients start complaining that a critical component is no longer working. When the new
release fails, operations phone will start ringing most likely late at night. The development
team is eventually notified so that they can start firefighting asap. The decision is made
to roll back the change almost immediately. After the rollback has been completed by
operations, development is tasked with analyzing and fixing the reason for the release
failure.
124
If we are honest and ask ourselves what is being felt at this point from everyone
involved, we may not like the answer.
Some are thinking who was the person who developed the piece of code that caused the
failure? How Did Quality Assurance miss the bug? Top management is asking for answers.
And although processes will look objectively at what happened, the finger pointing will
begin almost immediately. Many people inside the organization will start pointing fingers
most likely at Development and Quality Assurance. What this all means is that the
traditional blame game has begun inside the organization.
The sad truth is that blaming individuals for failures is the culture that most
organizations have not only embraced, but often promote. What we find is that inside
many organizations, this is a learned behavior that individuals learn from watching
others. The culture inside these organizations is one in which we accept criticism in the
face of failure. Over time this creates a culture inside organizations that embeds fear in
the unconscious mind and negates the ability to innovate.
What we discover is that as organizations grow over time they often move from a culture
where they have grown by allowing innovation because of the lack of fear, but then over
time the organizations implement policies and an increasing hierarchy that often
unknowingly serves to stifle innovation and grows fear inside the organization.
The move to the DevOps culture is often a move to try and regain the culture that was
pervasive during the early days of the organization. In DevOps, the concept of fail fast and
learn from it becomes ingrained in the culture and we expect people to make mistakes
and more importantly we accept them without the fear of reprisal.
In not only DevOps. But even in Lean it’s a tenant that we fully understand and accept
that imperfection happens and that learning from them should become accepted and the
norm. We deal with imperfection in DevOps by designing and embedding feedback loops
into processes so that when mistakes are made they can be identified early on in the cycle,
and they can be rectified before going any further. The teachings of Deming preach about
creating a culture that does not believe in blaming individuals, but instead focuses on
constantly improving processes so that the system identifies variances during the
development lifecycle and not at the end of the lifecycle. A DevOps culture strives to adopt
a blameless culture in which blame is not part of the process since we have true measures
and checks in place to capture issues in the developmental stages, and not in production.
By adopting a DevOps culture, we allow teams to work fearlessly, so that they can try new
125
experiments, and hopefully continuously improve the process. In short DevOps is about
people and not tooling, which is fully in line with the teachings of Deming.
When a DevOps culture has been instilled inside an organization we find that among
teams, there is a sense of shared responsibility where the entire team takes ownership
and works in a more collaborative manner.
An additional point to make about the blame game is that the conflict between
Operations and Development has been created by the organizations themselves over time.
These organizations in some ways thought that there should be an incentive for conflict
between Operations and Development as a sort of check and balance. The checks and
balances were seen as having Operations making sure the Development team was creating
software that would make efficient use of the hardware organizations invested in and that
someone was in some ways monitoring development to make they created software that
had just enough performance to satisfy customers and was secure enough to meet
minimal standards.
126
Creating Culture Change through Organizational Behavior
A common refrain we here in the DevOps movement is that people are more important
than tools and that DevOps is about culture change. What we don’t hear are any concrete
specifics about changing culture, instead we get bits and pieces of the culture puzzle. The
reality is that just like DevOps borrows from Lean, we must again borrow practices from
another scientifically based discipline called Organizational Behavior and its related
subfields of study.
The question of course is why do we need to understand Organizational Behavior in

order to help with the DevOps movement inside an organization? The answer goes back
to a statistic reported earlier in the book from Gartner showing ninety percent of
organizations who attempted DevOps without first tackling culture change are failing in
their efforts. What we discover is that without the kind of information that the study of
Organizational Behavior provides, organizations have no basis for following any one
DevOps course instead of another. The result is that these organizations often make
unsound decisions and wait for changes in the productivity promised by DevOps which
never materialize. Luckily, however, following basic scientific principles in Organizational
Behavior can provide the insight necessary to avoid making these sorts of mistakes.
It is often mentioned in DevOps that in order to understand DevOps, we need to look

back at Lean Manufacturing and specifically the Toyota Production System. Well, in order
to understand why we should employ Organizational Behavior we will also look back at
how companies like Toyota has successfully used Organizational Behavior in the past. In
the late 1980’s Toyota became one of the first Japanese owned companies to open up
manufacturing operations in the United States. Toyota realized from the beginning that
there were stark differences between American workers and Japanese workers, so they
employed Organizational Behavior techniques to help them better fit into the United states
from a corporate image level all the way down to understanding individual differences so
that they could put the right person in the right place. To this day, if the Lean practices
in place at various United States manufacturing facilities are compared to Japanese
facilities, you will find that the same principles/tools are used, they just are implemented
in different ways to better fit the differences in cultures.
127
What we will learn
The field of Organizational Behavior and its corresponding subfields are quite large and
would take extensive study and a number of books to fully master, we can instead discuss
a few concepts and how they can be applied to the DevOps movement.
The Mirror Image Fallacy

As has been pointed out already in the book that culture change is the most difficult
aspect of DevOps, in order to understand why, we need to be honest and look at the Mirror
Image Fallacy as an overriding reason why culture is so difficult to change in most
organizations. If you ask the majority of managers in most organizations who are looking
to move to DevOps about their culture and the buy in by their employees, they will almost
to a tee tell you that they have a great culture that probably does not need to change and
that almost all their employees will see the benefits of DevOps and buy in.
There are two reasons why management falsely believes that their culture is great and
that they will get buy for DevOps by the employees. The first issue is the Mirror Image
Fallacy which states that it is a false belief that all people are alike or that others share
one’s own abilities, beliefs, motives and predispositions. What this really means is that
inside these organizations there is not going to be complete buy in for DevOps by all
employees, in fact as was discussed in an earlier section there will be laggards who will
not only fight DevOps but may even try to sabotage the movement inside the
organizations. Management inside organizations and the innovators leading the DevOps
movement inside their organizations must be realistic from the start and acknowledge
that they are going to have to battle a number of employees and managers to get the
DevOps movement accomplished.
The other reason management falsely believes they don’t need to change culture is for
the simple reason that if they were to actually acknowledge their organization needed to
change culturally, they would view it as almost an indictment on themselves since they
are the management of the organizations and are viewed as setting the culture inside the
organization. Many managers would truly believe deep down that fingers would be pointed
at them if anyone were to bring up the need for culture change, so few, if any managers
are ever going to agree that the culture needs to be changed.
What all this means is that the ninety percent of organizations failing in their attempts
at DevOps are at some point going to have to acknowledge that, their organization is in
128
need of a culture change and that it is not going to be easy since some members of the
organization will not see the need for DevOps the same way they do.
Threats to Rational Decision Making

Once an organization has overcome the hurdle of recognizing that they do indeed need
to make some cultural changes in order to increase their chances of successfully moving
towards DevOps, a new threat can emerge that must also be recognized. While attempting
to implement DevOps the teams/groups/individuals involved will attempt to do so
through a Rational Decision Making Process, but what they will discover is that this is
neither easily nor commonly achieved in most organizations. The problem with the
Rational Decision Making Process is that it is not nearly as straightforward as it may seem
due to what are termed Decision Making Illusions.
Decision Making Illusions:
Loss Aversion Bias - Organizational Behavior research has shown that people have a slight
preference for sure outcomes over risky ones. What this means is that when organizations
are involved in their DevOps transformation, these organizations when faced with tough
decisions in terms of changes to their processes, deployment pipeline or tool selection may
weigh risk more heavily than possible gains even when the total value of each is equal.
You can often see examples of this when an organization determines that a critical piece
of their framework is obsolete and must make a determination whether to replace it or
attempt to create some type of work around. All too often these organizations will view
creating the work around as less risky, even though the replacement of the obsolete
framework piece has a larger possible gain with equal risk.
Escalation of Commitment - This illusion is seen when organizations making the move to
Agile and DevOps invest additional resources into a failing course of action that is not
justified by any foreseeable payoff. The Escalation of Commitment is seen in almost every
129
organization that has attempted Agile or DevOps and failed to make the necessary cultural
changes but will continue pumping resources and money into tools and consultants to
try and achieve either Agile or DevOps.
Understanding the Social Environment

One of the trends in the DevOps movement is for organizations to adopt some form of
a Dojo or Center for Excellence in order to help scale their DevOps movement. Dojos are
a concept introduced by Target where individuals and teams are taken to a location to
work for days, weeks or months with experts in the area of Agile, DevOps, Automation,
Tools, etc. etc. A number of organizations have attempted to follow this concept, but few
have asked the all-important question of why concepts like this have worked at places like
Target in order to better understand how to setup their Dojo or Center for Excellence. If
we take a look at Organizational Behavior we can better understand the underlying factors
behind Dojos and Centers for Excellence. The underlying reason that these concepts have
been successful in some organizations is that they have been able to manipulate their
Social Environment in a way that has spurred a positive reaction from their employees.
Functional Attraction - Is the idea that satisfaction in the workplace with other members
of their workplace comes about because these other people help one attain valued work
outcomes. When individuals are sent to these Dojos or Centers for Excellence they are
shown ways which not only are more productive, but, make their jobs easier.
Entity Attraction - We also need to realize that Dojos and Centers for Excellence are not
just individuals but are entities. Entity Attraction comes about when satisfaction with
others in the workplace is gained because these people share one’s fundamental values,
attitudes or philosophy. In short, the majority of employees inside most organizations
know there are better ways to accomplish their work and would love to try new methods,
techniques, tools and processes, but due to an organizational culture not in line with their
values, attitudes and philosophies they stagnate. When these same employees are taken
to Dojos or Centers of Excellence and allowed to work in an environment that is in line
with their values, attitudes and philosophies they often thrive.
Social Support - This is the idea that by having a surrounding environment in which
people are sympathetic and caring people can thrive. The vast majority of experts who
staff Dojos or Centers of Excellence have themselves seen the organization culture first
130
hand and experienced the same issues that the individuals sent there experience in the
typical day.
Buffering - This is the idea that certain positive factors in a person's environment can
limit the capacity of other factors to create dissatisfaction or stress. In a nutshell, Dojos
and Centers for Excellence serve as buffers in the social environment inside an
organization.
DevOps Targets the Symptoms of Stress

A possible side benefit of DevOps when successfully implemented may be its ability to
increase job satisfaction and reduce stress. How does DevOps help increase job
satisfaction and reduce stress you ask? From Organizational Behavioral research, we
know that a person's ability to handle dissatisfying or stressful work is greatly enhanced
when people feel they have the opportunity to participate. The term for individuals in the
workplace being able to participate is referred to as “voice”. We know from earlier sections
of the book that Lean and Deming preached that everyone in the organization needs to be
working towards common goals and that DevOps also is about breaking down Silos and
shifting ownership to teams where individuals can act as equal members.
Organizations also may receive additional benefits from DevOps ability to increase job
satisfaction and reduce stress. Research shows that when employees are able to
participate in decision making, this leads to less role conflict and role ambiguity, which
as a result causes less emotional stress and has the benefit to the organization of reducing
absenteeism and turnover.
Barriers to Effective Communication

Two of the critical elements of successfully making the cultural changes necessary to
allow for the adoption of DevOps discussed earlier were Vision and Having a Plan. Even
organizations who put together a quality Vision and Plan can still fail in their DevOps
efforts because they have failed to understand some common barriers to communication.
For any Vision or Plan to be accepted inside an organization the leaders of the DevOps
movement must be able to effectively communicate the Vision and Plan to the organization
as a whole by being aware of the organizational, interpersonal and personal factors that
can hinder their efforts. Below we will discuss some general barriers to communication.
131
Organization Factors - Spatial Arrangements - The physical space of the work place inside
organizations can inevitably affect communication. Research shows that people who work
closely together have more opportunities to interact and are likely to build lasting
relationships. So, if an organization is looking to build interpersonal relations, they may
want to closely look at the physical work environment they have built. Spatial
arrangement may also be a factor in the DevOps trend of shifting from outsourcing to
doing work inside the organization.
Role-Set Factors - The message the DevOps leaders should be attempting to communicate
should be two parts, an effort to inform and secondly an effort to persuade the
organization. All too often these leaders, specifically management fail to understand that
the credibility of the message source will largely determine whether the message is
internalized by the recipients. The credibility of the message sender refers to the degree in
which information provided is believable. Credibility has two important factors that
should be taken into consideration, first is the expertise of the message sources knowledge
of the topic. The second factor is trustworthiness that the recipient believes the
communicator has no hidden message. As has been discussed in earlier sections, if the
DevOps leaders lack the knowledge of the foundations behind DevOps and are unable to
provide a message backed by scientifically based facts, they are unlikely to find a receptive
audience.
Role-Organization Factors - In most organizations, people experience message overload

on a regular basis. This message overload means that incoming information needs to be
filtered. All the same factors that influence a person's perceptions affect communication
as well. This means that people tend to discount information that is not in accordance
with their beliefs. Messages that contradict people's beliefs may not be correctly
understood. We often see this in DevOps with the introduction of automated testing and
the perception of both Quality Assurance and Development.
Interpersonal Differences - When communicating with different functional units inside an

organization it's important to remember that each of these functional units has a different
frame of reference. It's not uncommon inside many organizations for functional units such
as development and product management to have their own jargon. Jargon is the
idiosyncratic use of language that is often useful among specialists but more often than
not inhibits their ability to communicate with non-specialists. One way to counter the
issue of specialized jargon by DevOps leaders is to implement Specification by Example,
which was discussed in an earlier section. Specification by Example acts as a common
language for all functional units.
132
DevOps Communication Roles
As the DevOps movement is initiated inside an organization by a group of innovators
and their possible management sponsor, it may be a smart decision by the group to
recognize special communication roles to try and deflect the problems sometimes created
from poor communication across roles in the organization. These specialized
communication roles have the sole purpose of improving the effectiveness of the groups
communication.
Gatekeeper - The Gatekeeper role is responsible for controlling the messages sent through
a specific communication channel. In DevOps, the Gatekeepers can be particularly
essential in preventing information overload from reducing the effectiveness of those in
upper level roles. Often the DevOps leaders will have a senior level management sponsor
who needs to be well informed in order to champion the group's message to others in
management, but the sponsor only needs certain pertinent information and not a
complete information dump. This is where a Gatekeeper who is familiar with effective
communication with management can be very effective.
Cosmopolitans - The role of the Cosmopolitan is important for the DevOps movement
because this is usually a person who has special knowledge the group needs, or they may
be affiliated with professional groups that allows them to keep up to date on specific
subjects or they have former contacts that gives them special expertise. In DevOps, this
may be as simple as a group member who has previous experience with infrastructure
tooling or even a member who has worked for an organization that was effective in moving
to Agile or DevOps.
Opinion Leaders - When putting together a DevOps group to try and push the DevOps
movement inside an organization, one of the first member the group should try and bring
on board is any Opinion Leaders they identify inside the organization. Opinion Leaders
are people inside the organization who have special access to the organization’s informal
channels of communication and therefore have the enhanced ability to influence others.
Dealing with Fear

To understand the concept of fear inside organizations and to have any chance of
driving it out, we need to factually understand what fear actually is. The reality inside
133
most organizations is that fear is simply negative politics, where individuals have figured
out that they can sometimes gain the political upper hand by attacking or blaming others.
By blaming others for negative outcomes, we make them scapegoats for the failures of
groups or the organization. By creating these scapegoats, the political players inside an
organization attempt to triumph over their perceived adversaries. Another way the political
players attempt to triumph is by denigrating or belittling other’s accomplishments.
How to Deal with Negative Politics:
Set Examples - It's critical that management recognizes the role negative politics plays
inside many organizations and works to set examples that discourage these political
games. Management must set out to make it clear that they will not tolerate deceit or dirty
tricks and most importantly they themselves must refuse to engage in politics. If
management makes it clear that political tactics are unacceptable, they will find that their
subordinates will be discouraged from engaging in negative politics as well.
Communicate Openly - DevOps teams must share all relevant information with coworkers
and the organization as a whole in order to alleviate destructive politics. Open
communications help ensure that everyone understands and accepts the vision and plan
moving forward. One of the surest ways to sink the DevOps movement inside an
organization is to employ it as a skunk works project.
Reduce the Uncertainty - When the DevOps leaders clarify their goals, tasks and
responsibilities they make it easier for all to assess people’s behaviors and make politics
hard to hide. Opening up the decision-making process helps make the decisions
understandable and discourages undercover politicking.
Manage Informal Coalitions and Cliques - Unfortunately inside many organizations we

find a number of informal coalitions and cliques who have the ability to quickly derail the
DevOps movement. It's important to not only identify these coalitions and cliques, the
DevOps leader must also influence the norms and beliefs that steer group behaviors in
order to ensure these employees continue to serve the organization's best interests. In
some cases, management may have to reassign some of these individuals in order to
abolish the groups politicking.
134
Confront Political Game Players - The DevOps leaders may find that even after their initial
attempts to discourage negative politics with certain individuals inside the organization,
they may need to call upon that individual's superior in order to curb that individual’s
political pursuits. In some cases, it may even become necessary to resort to disciplinary
measures to curb these individual’s political aspirations and even sometimes outright
dismissal.
Anticipate the Emergence of Negative Politics - DevOps leaders must be keenly aware from
the start of any circumstances that are conductive to politicking.
Resolving the Blame Game with Superordinate Goals

In the Blame Game section, earlier in the book we discussed the built-in conflict many
organizations have put in place pitting development against operations. This conflict
comes about through their conflicting incentive structures and differing views of software
development. It is important for organizations to understand that conflict requires
interdependence. This means that it is possible to manage or resolve this conflict by
restructuring the connections that tie these two groups together. In order to do this,
organizations can create superordinate goals, which involves identifying and pursuing a
set of performance targets that these groups can only achieve by working together. By
setting these superordinate goals, the organization causes the groups to tighten their
interdependence.
Want to do DevOps? Better Understand Organizational Development

At the heart of the DevOps movement is the basic idea that you are trying to change
the organization. Organizational Development is a planned approach to interpersonal,
group, intergroup and organization wide change that is comprehensive and long term and
is usually under the guidance of some change agent. In the case of DevOps, the change
agent is DevOps. It is also important to note that Organizational Development is also a
field of research that is specialized in developing and assessing specific change or
intervention methods.
135
The Five Features of Organizational Development:
An emphasis on Planned Change - Looking at the history of Organizational Development,

we see that the field evolved out of the need for a systematic, planned approach to
managing change inside organizations. What you see in the DevOps movement today are
attempts at change inside organizations with little to no planning, while being more
spontaneous or less methodical may be ok when we are trying new tools as part of say the
DevOps pipeline, this seldom works when we are attempting organization wide cultural
change.
Focus on Social-Psychological Factors - The field of Organizational Development is neither

focused on psychological (individuals) nor is it focused solely on sociological
(organizations), it instead incorporates a mix of both. This is important because
Organizational Development can stimulate change at almost any level. This is another
area where many DevOps leaders have failed to grasp the overall complexity of bringing
change to the organization. The leaders of the DevOps movement need to look at the
sociological environment of the organization and create a social environment that allows
for the psychological needs of the individuals to blossom.
Focus on Comprehensive Change - One of the tenets of Organizational Development is

that while its planners may have focused their intervention on a specific organizational
target, they also must keep in mind the effects of the change on the total system. This
brings up another mistake some DevOps movement leaders have made in the past, for
example the move to incorporate Unit Testing can have a far wider impact on the
organization then simply developers. In order to move forward with Unit Testing, we would
also want to gather buy in from Product Ownership who may have to accept a lower
velocity for a period of time while the Unit Testing is ramped up. In addition, there would
also need to be buy from the business, who in reality would be paying for the Unit Testing
incorporation.
A Long Range Orientation - Organizational Development is in complete lockstep with both

Lean, Deming and DevOps in that it is focused on change being a continuing process that
can sometimes take months or even years to bring about. What we too often see in the
DevOps movement today is management who are all too often faced with pressures for
quick, short term gains and will settle for a DevOps plan that simply yields stopgap
solutions.
136
Guidance by a Change Agent - Organizational Development interventions are designed,
implemented and assessed with the help of a change agent, which serves both as a catalyst
for change and as a source of information about the organizational Development process.
What we are seeing today is DevOps being the change agent and providing the catalyst for
change, but unfortunately it is all too often lacking the information about the
organizational development process. Organizational Development has shown that
successful organizational development does not grow out of an unguided, do-it-yourself
approach to organizational change. This is a critical concept to understand because what
we see today with the DevOps movement is almost a mirror image of what we have seen
in the past with the Agile movement, and that is organizations not following the
foundations of both movements and instead picking and choosing what they wish to
incorporate and having a belief that they know more than the founders and proponents
of these two movements.
Sources of Resistance to the Change Agent

There are three general categories of change that take place in most organizations and
whenever any of these three types of change are set in motion some level of resistance can
be expected. The basic human nature is for people to resist what they perceive as a threat
to the established way of doing things.
Three Categories of Change:
Adaptive Change - This type of change normally involves the reintroduction of a familiar
practice that has been used before in the organization or it was used someplace else in
the organizations. This type of change is marked by a lower potential for resistance and
has a lower level of uncertainty, cost and complexity.
Innovative Change - This type of change may introduce a practice that is new to the
organization but is widely used elsewhere in the industry. This type of change is marked
by a moderate potential for resistance and has a moderate level of uncertainty, cost and
complexity.
Revolutionary Change - This type of change is marked by the introduction of a practice

that is not only new to the organization, but the entire industry. This type of change is
137
marked by a high potential for resistance and has a high level of uncertainty, cost and
complexity. The introduction of DevOps would fall into the Revolutionary category.
Sources of Change Agent Resistance:
Self Interest - Change makes people fear that it will be difficult or impossible for them to
satisfy personal needs and desires at work. Individuals who exhibit Hero Cult tendencies
may fear DevOps takes away their autonomy for how they have coded in the past.
Fear of the Unknown - DevOps can cause fear of the unknown, because people inside the
organization simply do not know what to expect, this in turn causes anxiety and
resistance. Since people have no baseline to measure DevOps against they simply fear the
changes it may bring.
General Mistrust - This is an issue caused when people mistrust other’s intentions and
behavior, these people are generally suspicious of impending change and this in turn
causes secrecy and even deception. If you look around most organizations you will see a
cycle of new methods and practices that management has tried to implement and failed
at time after time, this failure has breed mistrust and suspicion inside the organization.
In order to counter this problem, it's critical that any DevOps movement have complete
transparency and a well laid out plan that shows progress and setbacks.
The Fear of Failure - Another cause of resistance to change seen in DevOps is the fear of
failure some people develop when they start doubting their personal competence. You
specifically see this when DevOps attempts to change familiar practices and it requires
the learning of new tools.
Differing Perceptions - In the workplace many individuals and groups having differing
perceptions and goals, so what may be good for one group may be viewed as bad for
another. Implementing DevOps is about breaking down silos and removing ownership of
tools and processes by these silos, so some silos may not view the removal of their
ownership of a tool/process positively, while another group may see this as a positive.
Loss of Status - The DevOps movement is about allowing teams to take ownership of
products and make decisions regarding the product; some management may be
apprehensive about their loss of decision making power over the product.
138
Social Disruption - The DevOps movement often brings about change that disturbs the
existing traditions or relationships within and between groups, this can often throw the
interpersonal and group dynamics into disequilibrium and cause resistance.
Peer Pressure - One of the simplest ways to view how Peer Pressure can result in resistance
to the DevOps movement is to closely watch the Laggards as they use peer pressure on
those not directly affected by the change, they will resist it nevertheless to try and protect
the interests of the Laggards.
Poor Timing or Tactlessness - The DevOps movement may meet resistance if it is

introduced at an inopportune time or in an insensitive manner.
Personality Conflict - Resistance can evolve out of simple Personality Conflict between
someone advocating the DevOps movement and someone directly affected by the change.
If the Blame Game inside an organization has already caused conflict among some
members of development and operations, the DevOps leaders may want to be careful
about including them on the same team or at least attempt to smooth over any hurt
feelings before bringing the members together on the team.
Bureaucratic Inertia - Resistance to change may simply be built into the very structure of
the organization.
139
Fighting Resistance with Forcefield Analysis
Wagner, John A, and John R. Hollenbeck. Management of Organizational Behavior. Prentice-Hall, 1995.
140
In order to set change in motion we must identify and overcome sources of resistance,
while at the same time encouraging and strengthening source of support. Forcefield
Analysis is a diagnostic method that depicts the array of forces for and against a particular
change in a graphic analysis. Forcefield Analysis is often used as a component of the
Organizational Development process. Forcefield Analysis is a tool many DevOps leaders
would find useful when trying to envision the situation surrounding their efforts to sell
DevOps to the organization.
To use Forcefield Analysis, two lines are drawn, one representing the organization’s
present situation (a solid horizontal line) and another line depicting the organization after
DevOps is implemented (a dashed horizontal line). Next, forces are identified that support
the DevOps change are drawn as arrows pushing in the direction of the DevOps change.
Forces seen as opposing DevOps are drawn as arrows in the opposite direction. The length
of the arrows represents the perceived strength of the force represented by the arrow
relative to the other forces in the forcefield.
Overcoming Forces of Resistance Identified in Forcefield Analysis:
Education and Communication - One method is to disseminate the information about the
what, why and how of DevOps through one on one discussions, group meetings or through
various digital media available inside the organization. This approach is best used when
DevOps is being undermined by a lack of information or where available information is
inaccurate.
Participation and Involvement - Research has shown that most successful DevOps
movements inside organizations has come from within. In keeping with this, involving
those affected by DevOps in its design and implementation can go a long way towards
lowering resistance.
Facilitation and Support - Another issue that many organizations have failed to grasp is
that DevOps requires increased training of existing employees and is not something that
can be outsourced. Organizations going through the DevOps movement should look to
provide job training and emotional support through concepts like Dojos or Centers of
Excellence.
141
Bargaining and Negotiation - In the case of the Blame Game, the DevOps leaders may find
they are working with resistant employees and need to bargain and negotiate with them
in order to get them on board. One way of accomplishing this is to work with them to set
superordinate goals to change their minds.
Hidden Persuasion - This involves using covert efforts and providing information on a
selective basis to get some people to support the DevOps movement is unfortunately
sometimes needed. The problem with doing this is that lead to future problems if people
have felt manipulated.
Explicit and Implicit Coercion - As a last resort, sometimes the threats of negative
consequences must be used to change the minds of resistant employees. This is
sometimes forced upon the DevOps leaders when they determine that an employee is not
only resistant to the DevOps movement, but is working to sabotage the efforts.
Organizational Culture Versus the Informal Organizational Culture

One of the problems that many DevOps movements inside organizations run into is
that they may actually attempt to make changes to the official culture of an organization
in order for DevOps to take off, but for some unknown reason to them the culture changes
do not succeed. The DevOps leaders may be puzzled as to why they weren’t successful,
the problem is that they may have not understood that inside every organization is an
informal organization with its own culture. An informal organization is the unofficial rules,
procedures, interconnections and culture that develops as employees make spontaneous
unauthorized changes in the way things are done inside the organization. At the heart of
this informal organization is what is termed the “grapevine”, this is the unofficial
communication network within the organization by which employee’s trade gossip and
rumors about their jobs, their coworkers, and the organization. The DevOps leaders inside
an organization must be fully aware of this informal organization that is created as day to
day adjustments to the formal way of doing things creates a culture of attitudes and
understandings that are shared among coworkers. It is also quite common for the laggards
who may oppose or attempt to sabotage the DevOps movement to use this informal
organization to their advantage, specifically the grapevine.
142
The Elements of Organizational Culture
Inside the culture of every organization is a collection of fundamental norms and values
that serve to shape employee’s behaviors and help them understand the surrounding
organization. Being aware of and understanding these norms and values is extremely
important for the DevOps leaders, since it is these norms and values which must be
changed in order for the culture to be changed and DevOps to have any chance of success.
What are these fundamental norms and values and how are they expressed. There are
specific surface elements of the culture that help employees interpret everyday events
inside the organization and are the principal means by which cultural norms and value
are communicated from person to person. The first set of surface elements are
Ceremonies, Rites and Rituals which serve to reinforce specific norms and values by
demonstrating their worth through special events. The next set of elements are Stories
and Myths, which exemplify norms and values by encompassing them in memorable
narratives. The next elements are the exploits of Heroes and Superstars, these are
organization employees who show the results of adhering to the organizations norms and
values. Lastly, we have Symbols and Language, which serve as constant reminders of
important norms and values. Every group leading the DevOps movement should be keenly
aware of the specific surface elements inside their organization in order to try and
change/shape them in order to try and change the underlying culture inside the
organization.
Cross Cultural Communication

In this day and age, it is almost impossible to find any organization that is not made
up of employees from several different countries with different societal cultures.
Unfortunately, very few organizations spend any time investigating how these various
cultures influence the organizational culture. The reality is that the behavior in an
organization is also shaped by social information residing in the “national culture” of
societal norms and values that is part of the organization’s environment. It is highly
suggested that any DevOps movement inside an organization take into account the
possible cross-cultural differences and how they may affect the DevOps movement.
143
Let’s take a look at Dolphin now
Dolphin Explores their Problem’s
As we turn our attention to Dolphin, we find the CEO of Dolphin sitting in her office
contemplating how the company has gotten itself into the current situation it finds itself
in. While attempting to make sense of what has gone wrong, she thinks back to how she
led Dolphin in the early days and then watched as the company matured and continued
to grow, back then she felt sure of the direction she wanted to take the company in, but
these days she does not feel that same sense of confidence in the decisions she must
make. Our CEO decides that she wishes to meet with an old mentor of hers and use him
as a sounding board for the decisions she must make.
The CEO calls one of her old professors who has served as her mentor since her days
taking his Operations class as part of her MBA program. The professor is glad to hear
from her and would be glad to meet with her and talk about the looming decisions she
must make. A few days later the CEO meets her old professor and she begins by explaining
the predicament that Dolphin finds itself in. The CEO describes how the latest numbers
are showing that the new competitors in their vertical have not only been rapidly gaining
market share but are actually taking market share away from Dolphin. In addition, she
shares the research that Dolphin’s marketing team has gathered that shows among other
things, customers do not find Dolphin’s products and services to be high quality and that
Dolphins technology offerings do not seem to be keeping up with changing demographics.
The CEO then explains that she is now unsure of the direction forward, since it was
Dolphin that was seen for many years as a leading-edge technology player in the vertical
and that she can’t seem to put her finger on what has happened to the quality of Dolphins
products and services. The CEO explains that she has some definite suspicions as to what
has gone wrong, but that she does not feel that same sense of confidence in which
direction to take Dolphin in that she has over the years.
The professor takes in his protegees words and then asks her if she has gone back to
some of the fundamentals that she learned in the MBA program. She answers that she
has always relied on the knowledge she gained in the program, but this time is not sure
144
where to look for the answers. The professor explains that he would suggest that the first
thing she does is find a way to gather hard evidence of exactly what is happening internally
and externally. He points out that in his Operations class he teaches that you should
always make decision based on cold hard facts, not suspicions or best guesses. He next
explains that the answer to many of the questions are already available to her, and that
the source of this information is her employees. He strongly suggests that she seek the
answers to what has gone wrong and how it could be fixed, by simply talking with the
employees and listening to them. He suggests determining the questions that need to be
asked and then finding a way to ask them, it could be as simple as one on one talks, group
meetings or even surveys. The last suggestion the professor has is to go back to the roots
of her Operations management class and rely on the teachings of individuals such as Dr.
Deming.
Breakdown of Dolphin’s Problems:

1.
2.
3.
4.
145
Learning Point(s):
Denialism
“Motivated Reasoning.” Psychology Today, Sussex Publishers,

www.psychologytoday.com/basics/motivated-reasoning.
At this point, you may be asking yourself with everything we have covered so far in the
book, shouldn’t it be obvious to every organization that they should be making the
changes necessary to move too DevOps? You would think with all of the scientific evidence
available from sources such as Gartner, Xebia labs and DORA to just name a few, that it
would be overwhelming evident to organizations why they need DevOps and why it needs
to be done correctly.
The truth is that the evidence is overwhelming, but much like today’s current debate
about Global Warming, there are enough individuals who display what is termed
“denialism” to thwart the efforts in most organizations. First, we need to understand what
denialism is, it is part of the psychology of human behavior, where a person chooses to
146
deny reality, as a way to avoid a psychologically uncomfortable truth (we will discuss these
truths below). We need to understand that denialism is an essentially irrational action
that withholds the validation of a process, methodology or concept by the person refusing
to accept an empirically verifiable reality.
So, what causes this denialism in some individuals? Denialism is caused by the
tendency toward “motivated reasoning” in some individuals and groups. This motivated
reasoning helps explain why some individuals and groups become so polarized over new
concepts such as Agile or DevOps, even when the evidence is so unequivocal about their
benefit. It would seem that expecting people to be convinced by the facts flies in the face
of, you know, the facts.
The theory of motivated reasoning builds on a key insight of modern neuroscience,

where reasoning is actually suffused with emotion (or what researchers often call “affect”).
Not only are the two inseparable, but our positive or negative feelings about change
through initiatives like Agile or DevOps arise much more rapidly than our conscious
thoughts, in a matter of milliseconds. Motivated reasoning is a “basic human survival
instinct”. We push threatening information away; such as Agile or DevOps and we pull
friendly information close. In other words, we apply traditional fight-or-flight reflexes that
are normally associated with predators, and instead apply it to data itself.
The first truth that individuals refuse to accept in many organizations is “uniqueness”.
One of the most common comments you hear from a small set of individuals inside the
majority of organizations when the concept of either Agile or DevOps is introduced, is that
Agile or DevOps will not work as explained because of how unique their architecture or
products are. This is a false argument in almost every organization worldwide with
absolutely no backing evidence by any of the denialist. The reality is that in the case of
their unique architecture, the truth is that they either are using outdated technologies,
had a poor design in the first place or have been lax in keeping the various architecture
pieces current.
The other common argument the denialist use is that the products they release can’t
be developed in an Agile or DevOps fashion. One argument for this that is commonly heard
is that they release to a large number of unique clients who have different variations of
the product, so it would be impossible to do things like Unit Testing since each client
would need a set of Unit tests for their particular variation of the product. Again, this is
simply a false argument. This is an issue that could be solved in a number of ways, such
as through using Specification by Example to identify only the critical functionality that
147
needs to be tested and not trying to cover every conceivable scenario as the denialist often
point out. Another cure could be designing a product with reusability in mind, instead of
trying to design a unique variation for every client as many organizations mistakenly do.
The truth is that the denialist in the organization are part of the laggard’s group we
have discussed throughout the book. The leaders of the DevOps movement inside the
organization must be prepared for the uniqueness argument to come up almost right away
when either Agile or DevOps are pitched to the organization. The DevOps leaders must as
part of their plan, be able to paint a clear picture to the organization that demonstrates
that the organization is not unique, and that the uniqueness argument is not valid.
Listening
Inside many organizations who have struggled with Agile or DevOps or are thinking
about making the move to either methodology, you will actually find individuals who either
have a basic grasp of either methodology, have an understanding of some key
component(s) of either methodology or are well versed on what is involved in either
methodology. The problem is that these individuals are all too often not listened to inside
their organization for a number of factors.
148
One of the key factors that silences these individuals is what is termed an “emphasis
on titles”. All too often, the individuals who have the knowledge needed to help with either
Agile or DevOps are quickly silenced by other individuals who title wise are their superiors.
The problem is that often these individuals with the titles are not superior to the
underlings in knowledge, skills or leadership abilities. All too often many of these
individuals in the organization will use their titles to silence ideas and concepts in order
to protect their title and are not looking out for the organizations greater good.
Another problem often seen in organizations involves not listening to

individuals/teams/groups who identify where additional planning is needed which is
counter to the original plan that was developed. This is what is termed “favoring a plan
over planning”. What this means is that often inside organizations a group of individuals
will come up with a detailed plan for everything from developing an application, to
introducing new tooling to changes to the architecture, but when the individuals who are
responsible for implementing the plan start the actual implementation and realize some
change to the plan needs to be made, they are often ignored, and the original plan just
sits and collects dust. At the heart of both Agile and DevOps are the idea that continuous
planning should be happening as individuals/teams/groups learn and adjust.
Luckily for us, other industries have tackled the idea of listening to everyone in the
organization and have even come up with some very effective approaches for doing so.
One of these is General Electrics “work-out” approach.
The idea of including everyone in the process of improvement is one of the key principles
of TQM, Lean and now DevOps. There are numerous ways being suggested in DevOps
articles, blogs and books to involve everyone and create a continuous improvement
culture. We suggest that instead of trying to re-invent the wheel, many of these
organizations should consider tried and tested approaches, such as the ‘Work-Out’
approach that originated at General Electric. This approach allegedly was developed by
Jack Welch, the then CEO of GE. The reasoning behind developing this approach was the
recognition that employees are an important source of new and creative ideas, and an
instrument for creating an environment that pushes towards a relentless, endless
companywide search for better ways to do everything.
The Work-Out program was seen as a way to reduce the bureaucracy typically
associated with improvement and give every employee, from managers to factory workers,
an opportunity to influence and improve GE’s day-to-day operations. According to Jack
149
Welch, Work-Out was meant to help people stop wrestling with the boundaries and
idiocies that grow in large organizations. Most of us are familiar with those idiocies, too
many approvals, duplication, shadow responsibilities, politics and waste.
Work-Out is credited with turning GE upside down, so that the workers told the bosses
what to do. That has forever changed the way people behave at the company. Work-Out
is also designed to reduce, and ultimately eliminate all of the wasted hours and energy
that organizations like GE typically expend in performing day-to-day operations.
Work-Out typically has a broad series of activities implied within the approach:
● Staff, key stakeholders and the responsible manager hold an off-site meeting away from
the operation.
● During the meeting, the manager gives the group the responsibility to solve a problem
or set of problems shared by the group, but which are ultimately the manager’s
responsibility.
● After the manager leaves, the group spends time (possibly multiple days) working on
developing solutions to the problems, sometimes using outside facilitators.
● At the end of the meeting, the responsible manager (and sometimes the manager’s boss)
rejoins the group to be presented with the recommendations.
● The manager can respond in three ways to each recommendation; ‘yes’, ‘no’ or ‘I have to
consider it more’. If it is the last response the manager must clarify what further issues
must be considered and how and when the decision will be made.
While Work-Out programs can add expenses; outside facilitators, off-site facilities and
the payroll costs of a sizeable group of people meeting away from work, even without
considering the potential disruption to everyday activities. These expenses need to be
weighed against the most important implications of adopting Work-Out, cultural change
(These benefits can be seen in approaches such as Target’s Dojo’s). In its purest form,
Work-Out reinforces an underlying culture of fast problem-solving. It also relies on full
and near universal employee involvement and empowerment together with direct dialogue
between managers and their subordinates.
150
What distinguishes the Work-Out approach from the many other types of group-based
problem-solving is fast decision-making and the idea that managers must respond
immediately and decisively to team suggestions. An additional way that approaches such
as Work-Out influence culture, is the acknowledgment at GE that resistance to the
process or outcome is not tolerated and that obstructing the efforts of the Work-Out
process is a career-limiting move.
Slack, Nigel, et al. Operations Management. Pearson, 2017.
ChatOps
ChatOps was designed to help connect individuals, teams, bots, and delivery pipeline
tools in an automated and transparent workflow, allowing everyone to see the complete
status of current work and systems. This transparency reinforces the feedback loop and
boosts collaboration.
151
ChatOps tools allow users access to important information directly from their chat
window, dramatically reducing the need for context switching. If you already have a chat
window open in listening mode, then it will help to connect it with the other tools and
platforms with which you have already been working. In addition, ChatOps provides user-
specific commands that feel intuitive because people can send commands to the chat
robots just as they would to any individual or team.
DevOps teams use comprehensive sets of online chat rooms and bots to drive delivery
activity without having to go and ask for help or have a meeting to plan for it. DevOps
teams put the chats at the center of their teamwork, essentially making it the real-time
operations center.
This new communication ability makes it easy to share and collaborate in real-time,
and the result is more effective DevOps.
Dolphin Looks for Answers
The CEO of Dolphin returns from her meeting with the professor and contemplates
what he has suggested. She comes to the conclusion, that he is correct and that she needs
to gather scientific evidence in order to make educated decisions on the path forward for
Dolphin. The CEO calls a meeting of her entire management team and explains to them
what she is thinking and how she believes they should proceed forward.
The first decision she has made is to call in an outside business consultancy firm that
they have worked with in the past and have built a trusted relationship with. The firm will
be coming in to do a SWOT (Strengths, Weaknesses, Opportunities and Threats) analysis
in order for the Dolphin management team to gain a clearer picture of what they face both
internally and externally. The managers are instructed to make themselves and their
employees available to the consultants when they come in and to answer any questions
they have or provide them with information they request.
The second decision she has made is to work with a survey company who will be putting
together an employee survey to gather information from the employees as to what is wrong
and what we could be doing to improve. The CEO points out that she will select a set of
152
managers to work with herself and the survey company to create a survey that asks the
pertinent questions they need answers to. She also points out that it’s critical that it be
communicated to everyone that they should feel free to answer the questions honestly
without any threat of retaliation.
Third, the CEO has also hired a transformation company who specializes in Agile and
DevOps to come in and give an assessment to members of the organization to attempt to
discover the pieces of the puzzle they are missing and need help with.
Lastly, the CEO emphasizes to the managers the same points the professor made to
her, they are going to make informed decisions based on evidence and that once they
finish gathering the evidence they will then put together a plan and that the plan will be
based on tried and true business fundamentals.
Breakdown of Dolphin’s Answers:

1.
2.
3.
4.
153
Learning Point(s):
SWOT Analysis
The problem of getting complete buy in from the entire organization for an undertaking
such as DevOps is extremely difficult. The problem is that this really shouldn’t be that
difficult given the very real threats and opportunities that DevOps encompasses for every
organization. Part of the problem is that most DevOps leaders have not sufficiently used
traditional business tools available to them in order to help sell the strategic reasons why
the organization must embrace DevOps. The way in which this should be done is through
the use of a “situational analysis” or as its better known as a “SWOT Analysis”.
To perform a SWOT analysis, the DevOps leaders should first look at the organizations
“strengths”, which are characteristics of the organization that can help it achieve its
strategic goals. As part of looking at the organizations strengths the DevOps leaders
154
should also point out any organization strengths in terms of helping adopt DevOps. In
terms of DevOps strengths, these could include having several DevOps pipeline
components in place, having a highly skilled workforce in place or already having a culture
in place that allows experimentation without fear.
The next area is an assessment of the organizations “weaknesses”. The weaknesses are
the organizations characteristics that could possibly block its progress towards strategic
goals and the DevOps movement. Some common weaknesses in terms of DevOps would
include, having a culture of fear, not having a highly skilled workforce, having processes
that do not adequately research alternatives or a lack of essential management talent.
The situational analysis should next focus on the conditions outside of the organization,
which should be the main selling point of DevOps inside the organization. The first
condition is “opportunities” offered by DevOps that would help it meet or as DevOps as
shown exceed its strategic goals. In the opportunities section of the analysis, the DevOps
leaders need to clearly make their case as to why DevOps would provide the organization
with opportunities to meet or exceed strategic goals that their competitors who fail to
adopt or attempt DevOps would not be available to them.
The last section of the analysis and of critical importance to selling DevOps, is the
“threats” that may prevent the organization from meeting strategic goals. The DevOps
leaders need to highlight to the organization the threats DevOps presents to the
organization if it fails to move to DevOps or fails in its attempt. These threats would
include a competitor(s) moving to DevOps and the specific advantages they would then
possess, a client possibly adopting DevOps and deciding they no longer need to outsource
the development of software your organization provides or lastly an organization outside
your business vertical adopting DevOps and thus increasing their development capacity
and then looking for new markets, such as your organizations market vertical.
Once the SWOT analysis has been developed and the DevOps leaders have polished it
up, then it should be presented not to just IT management, but the entire executive
management team and business units so that it is abundantly clear to the organization
as a whole why DevOps must be undertaken from a strategic management perspective.
155
Value Proposition Cards
A common problem in the world of software development is bringing Technical Debt to

the forefront so that organizations can clearly see why they need to make the move towards
Agile and DevOps. The issue is, how do we get everyone in the organization on board?
Even when you explain to the business side that research has shown that organizations
of all types and sizes are having to spend 70% to 80% of their entire Software Development
budget simply maintaining existing code, they don’t believe this includes them or they
simply don’t fully understand the issue. The problem is that they see the requests they
make to Software Development moving through the Software Development process and
believe this is all new work being created for their customers.
On the other hand, Software Development generally has a suspicion that they are
probably taking on Technical Debt, but due to lack of time, resources or simply the lack
of will they ignore it. Even when some Software Development groups are introduced to
tools such as SonarQube and its Technical Debt plugin they often times choose to ignore
it or are scared to see what it will show. The truth be told, you really can’t blame Software
Development for wishing this subject would stay buried, because they are often times
overwhelmed with their current work load and the last thing they want is perceived
additional work.
156
I think a major reason for Technical Debt not having the focus it should have by all
organizations is the lack of understanding of what the most important ROI in terms of
their software products is. The most import ROI is life span of a product, why is this you
ask? When we develop new software initially we have costs associated with resources to
develop it, infrastructure, marketing etc. etc.… and then we have continuing costs for
maintenance, additional features, infrastructure, etc., etc. The only way to recoup these
costs and have an acceptable ROI is for the product to have a long enough life span. We
then must take into consideration what is one of the biggest factors in the life span of a
product, its Technical Debt.
So how can we go about bringing Technical Debt to the forefront so that an organization
can properly prioritize it and adopt remediation processes to deal with it (Agile and
DevOps)? One possible way is through the use of “Value Proposition Cards”. In order to
explain Value Proposition Cards, we are going to use Kanban in our example. When we
create new cards for our Kanban board we would now start marking these cards as one
of two types, a “value proposition card or a failure proposition card”. So, what constitutes
a value proposition card or a failure proposition card? Well a value proposition card would
be any work that we can clearly identify as adding say a new feature (cannot be a re-do of
an existing feature), a new application or say framework changes that are not being done
to fix Technical Debt or limiting factors of the existing framework. Failure Proposition
cards include obvious things like bug fixes, feature changes due to not meeting the
customer’s needs, Technical Debt or framework limitations. On top of marking the cards
as Failure Proposition cards we would also do some root cause analysis, so that we could
categorize why the card is marked as a failure proposition card, in order to justify why it's
considered a failure.
We wouldn’t recommend going to deep into the root cause analysis, we just want to be
able to explain why its marked as a failure card and have justification when the
organization objects. If we were to also say, make the Value Proposition cards blue and
the Failure Proposition cards red and laid them out on a Kanban board, what do you think
your organization's board would look like?
157
Using Survey Feedback in DevOps to Stimulate Information Sharing
An important tool that should be in the toolbelt for all DevOps leaders is the use of
Surveys. DevOps leaders can use surveys to help initially highlight cultural, process and
tool impediments to DevOps. During the DevOps implementation, surveys can be used to
assess progress and surveys finally can be used to monitor the continual improvement
progress.
Using survey feedback is one of the Organizational Development intervention

techniques used for bringing about culture change throughout the entire organization, by
stimulating information sharing through the entire organization. It should be noted that
survey feedback is very different than the traditional questionnaire method of gathering
information.
In order to effectively stimulate information sharing throughout the organization by

using survey feedback as an intervention technique in Organizational Development, you
need to proceed in four stages as outlined below.
158
The first stage involves the DevOps leaders (change agent) engaging in preliminary
planning, deciding such questions as who should be surveyed and what questions should
be asked. It is ok for other organizational members to participate in this stage if they have
needed expertise, knowledge or their opinion is needed. In this stage, it may also be
necessary to involve an outside resource with knowledge of developing surveys and how
to statistically interpret them and follow up as needed.
The second stage involves the DevOps leaders and others involved in administering the
survey questionnaire to all the organizational members selected.
The third stage involves the DevOps leaders categorizing and summarize the data. After
the data is collected, the DevOps leaders present the findings to the organization through
group meetings so that everyone knows the results.
The last stage is for the teams/groups that received the information to hold meetings
to discuss the results. The DevOps leaders then work with these teams/groups to interpret
the data, help diagnose changes needed and help plan for needed changes. The DevOps
leaders are in fact acting as “process consultants” during the discussions.
Dolphin Reviews the Evidence
The CEO once again sets up a meeting with her entire management team to review the
evidence that has been gathered and filtered from the Assessment, SWOT analysis and
employee survey. The CEO starts out the meeting by discussing the general findings of
the SWOT analysis and survey, she says that there has been almost complete correlation
between the findings of the two. She also believes that the evidence has painted a very
clear picture of the path that Dolphin needs to go in.
The first order of business is the review of the SWOT analysis, the CFO gives the review
to the management team. He starts out by showing the complete SWOT analysis that was
put together by the consulting firm and briefly covers that, but then he turns to the critical
information that was gathered in the SWOT. The first area that they cover is a weakness,
the analysis has determined that the competition has gained an advantage over Dolphin
159
in the area of mobile applications. Currently Dolphin only makes it web based applications
available via mobile, while the competition has created a number of native mobile
applications that have caught on with the Millennial demographic group. The analysis has
determined this is a major weakness that must be addressed in the very near term.
The next major issue identified in the SWOT is a threat identified in the new
competitor’s due to them being able to develop and release new software at rate far quicker
than Dolphin. In addition, this software was also determined to be of a higher perceived
quality then Dolphin’s. It was determined that the rate at which these competitors are
able to release new software has the potential to not only put Dolphin out of business in
the next few years, but any other competitor in the vertical who cannot match their speed
and quality. A number of managers quickly ask the question of how these competitors are
able to release so quickly and with high quality, a couple of IT managers make the point
that they have actually heard some of these competitors are just releasing the software
straight to the customers and not doing any testing and that it’s the customers who are
doing the actual testing that is needed. The CFO explains that the survey of the employees
has actually provided the information to explain why the competitors are able to move so
quickly.
The CTO is the next to present and he covers the key findings of the employee survey.
The CTO explains that he is going to cover the key themes that were identified in the
survey. The first theme identified was that there is too much bureaucracy and lack of
departments working together. The survey results clearly showed a frustration by the
employees with the lack of communication among departments and the levels of
bureaucracy they must go through to get anything accomplished.
The next theme that was identified was the overwhelming suggestions that Dolphin
needed to move to the Agile methodology, the survey showed the current development
methodology was resulting in continuous delays, frustration and causing many of the
quality issues at Dolphin. The CTO at this point read a number of the specific responses
they gathered that lead to this theme being identified. The CTO described the next
identified theme and explained that this one helped them understand why the competitors
were able to release software more quickly and with higher perceived quality. The theme
identified was that Dolphin needed to start moving in the direction of Continuous
Integration, Continuous Deployment/Delivery and possibly something called DevOps. The
CTO explained that he was not going to try and give a detailed description of what these
were, but the managers should think of them as automating much of the current
development process, changing the development processes and a culture shift.
160
The CTO also discusses the Assessment results and simply states that the findings
were extremely eye opening and disappointing when viewing them in terms of looking at
the organizational culture, continuous improvement efforts and learning among others.
The CTO points out that on the bright side, the Assessment has clearly identified the areas
of weakness for the organization.
The CEO finishes the meeting by covering the next steps that Dolphin will take. The
CEO states that she has tasked the CTO with finding a consulting firm that can come in
and give the management team a concise high level presentation of both the Agile
methodology and these Continuous Integration, Continuous Delivery and DevOps terms
that were discussed. The CEO also explains that they will hold off on the move forward on
the mobile development until they are better able to understand Agile and other pieces,
since they would have direct impact on Dolphins development processes.
Breakdown of Evidence Review:

1.
2.
3.
4.
161
Learning Point(s):
162
DevOps Business Advantages
163
Kim, Gene. “Why Everyone Needs DevOps Now - Gene Kim.” LinkedIn SlideShare, 13 Oct. 2014,
www.slideshare.net/Dynatrace/why-everyone-needs-devops-now-gene-kim.
“5 KPIs That Make the Case for DevOps.” Puppet, 19 July 2013, puppet.com/blog/5-kpis-make-case-
for-devops.
Bittner Follow, Kurt, and Andrew Phillips. “Building a Compelling Business Case for Continuous
Delivery.” LinkedIn SlideShare, 7 Oct. 2015, www.slideshare.net/xebialabs/building-a-compelling-
business-case-for-continuous-delivery?qid=166bee73-4beb-4a13-98f5
9b567c493419&v=&b=&from_search=2.
So far in the book we have discussed that DevOps delivers clear advantages (at a
theoretical level) to the organizations who effectively adopt it versus their competitors who
fail in their attempts or disregard it at their own peril, but we have not yet discussed the
specific advantages DevOps has proven. When we discuss the concrete advantages that
successful DevOps organizations have experienced, we must make it extremely clear that
these organizations have fully adopted DevOps and not just cherry picked a few concepts
or implemented a few tools, they have instead fully embraced the cultural changes
necessary.
One of the clear advantages DevOps organizations have experienced is the ability to cut
cycle time by almost ninety percent versus their competitors. So, what exactly does
reducing cycle time mean and why is it such an advantage to these organizations. Cycle
time is the amount of time from the inception of a new idea to the new application or
feature being developed and then finally being delivered to the end user. In the majority
of organizations, the cycle time for idea generation to actual delivery to end users is
normally measured in weeks, months even years, so imagine an organization that could
cut this time by ninety percent. This means these organizations not only get the new
application/feature in front of the end user far quicker, but they are able to make sure
the application/feature works as expected through feedback loops, monitoring and A/B
testing and they have probably also delivered additional applications/features before the
traditional organization has even delivered their version of the application/feature.
The one question that almost everyone asks who is new to DevOps when they hear
about the dramatic cut in cycle time is that the quality obviously must be less than in
traditional development methodologies. The quality actuality is far better then what
traditional software releases experience. Organizations practicing DevOps on average
experience fifty percent fewer failures in their delivered software than traditional software
deliveries experience and maybe more importantly they can recover from a failure almost
twelve times as fast as their non-DevOps competitors.
164
The organizations practicing DevOps also on average release software almost thirty
times as frequently as non-DevOps organizations, which in reality means they have
increased their capacity without necessarily increasing their resources, which equates to
an almost staggering fifty percent increase in productivity.
So, what does all this paint a picture of, it paints a picture of organizations who have
successfully put into place what we would term prerequisites for what should be their
ultimate goal for adopting Agile and DevOps and that is to shrink the environmental gap
between their products and services and the customer and market needs they serve.
Organizations who have truly made the leap to Agile and DevOps can be seen as having
the ability to hear a pin drop when their customers or markets start to signal a change
versus the bundle of dynamite that customers and markets must blow up in order to
signal those organizations who have yet to move to Agile and DevOps.
Become a Story Teller
When trying to persuade others in the organization to get involved with the DevOps
movement, or attempting to sell the business side or management on DevOps it would
165
seem to many that simply laying out the scientifically measured benefits would be more
than enough to convince everyone of the need to adopt DevOps. Well, the reality is that
just laying out the DevOps benefits simply does not convince and more importantly
motivate everyone to support and get involved in the efforts.
Yes, by laying out the facts to the business side or management you will often get a nod
of agreement that this sounds like a good idea and that the business side or management
is on board. What this nod of agreement means in many cases, is that yes, the business
or management is fine with IT undertaking this DevOps undertaking as long as it doesn’t
in any way slow down the amount of work being released. In other words, these benefits
sound great to others, but they are looking short term and that means DevOps better not
interfere with any current or proposed work.
So, how do you attempt to not only sell others on the benefits of DevOps, but truly
garner their buy in? One method may be to use stories as an effective way of persuading
others. Research has shown that effective storytelling is a very effective tool when seeking
to persuade others to one’s point of view. The research has also shown that opinion-based
rhetoric is often more polarizing than persuasive, and that statistics often go in one ear
and out the other. But a careful blending of rhetoric and facts, woven into the right story,
can change minds.
So, what kinds of stories should one use when trying to persuade others to join the
DevOps movement inside one’s organization. There are two types of stories individuals
and teams should use when attempting to persuade other to join the DevOps movement.
The first, is possibly using stories from other organizations who have gone through great
endeavors to implement DevOps in order to highlight that it is possible and that these
organizations faced the same challenges your organization is facing. The second type of
story is one in which you describe the successes that have happened during your own
organizations DevOps endeavor, no matter how small the success was.
Next, we will discuss what constitutes a persuasive story:
Know your message - A good persuasive story should have more than a point of view; they
need to have a message. As such they are tools of persuasion.
Find the right example - If we want to demonstrate the benefits of DevOps, then the story
should explain how individual and teams would benefit.
166
Weave your narrative – The best stories use real-life examples. An example is how a team
adopted Agile or DevOps new and achieved improved results. Tie this to a narrative by
following strong story structure. Describe the situation. Talk about what happened. Close
with the benefits pitch.
Convey passion – Don’t be over the top, but you do need to demonstrate your conviction.
Do this through your choice of words, ones that draw pictures. And do it through your
delivery, raising your voice on a key point, pausing for emphasis and following through
with well-paced flow.
Support with facts – Just because you use a narrative approach doesn’t mean you can’t
use facts. Add them into your narrative, which can be as simple as beginning or ending
your story with them.
Dolphin Learns about Agile and DevOps

The CTO discusses the move towards Agile, CI, CD and DevOps at lunch with a couple
of his managers, they all state the same basic position, that they are somewhat familiar
with these methods, but don’t feel comfortable enough with them to lead the movement
at Dolphin. The CTO believes that the move forward is critical to the survival of Dolphin
and that he must find the correct partner company who can explain what these
methodologies are, what all is involved and possibly help lead Dolphin forward if needed.
The CTO decides he will make a few calls to some trusted industry contacts and pick their
brains regarding possible partners to work with.
During the CTO’s calls with his industry contacts, he discovers a wealth of useful
information along with some suggestions for who to contact. Some common themes come
out of the phone calls, the first and possibly most important is to understand that it’s all
about culture change and that if you are not willing to make the corresponding culture
changes, then you will probably not be successful with Agile or any of the other
methodologies. Another theme that comes out of the calls is that these methodologies are
167
all interrelated, many of the contacts discussed downstream and upstream Agile. Lastly,
everyone points out the need to take a close look at how IT is viewed in the organization
and to even look at exactly what type of organization you really are. A couple of contacts
pointed out that their own organizations had to come to terms with the fact that they were
really a technology company who happened to compete in a market that was traditionally
not seen as being made up of technology companies.
Another point a few contacts made was that DevOps should be looked at as an extension
of Agile applied to the entire company. The contacts also suggested that Dolphin not look
towards any of the tool vendors offering Agile and DevOps consulting because they will try
and get you believe that these methods are about tools and if you would simply buy their
tool set you could be doing Agile, CI, CD and DevOps.
The CTO takes this information and sets up a couple of meetings with the consultancy
firms that were suggested. During the meetings, the CTO specifically looks for the firms
to discuss the themes his contacts had mentioned. The CTO decides to choose a local
smaller firm that not only covered all of the themes he was hoping they would mention,
but actually went into even more detail. The CTO explains that he is really looking for a
firm that can be a partner with Dolphin during this journey, the consultants explain that
they are Coaches first and that they can be there at every step of the journey, but it must
be understood that at some point Dolphin must take over, otherwise they feel they will
have failed as coaches. This is the response the CTO had been hoping to hear. The CTO
sets up a date and time for the firm to come in and give a presentation to the management
covering Agile, CI, CD and DevOps.
168
Breakdown of what Dolphin Learned:
1.
2.
3.
4.
Learning Point(s):
DevOps as an Extension of Agile
Earlier in the book we discussed our opinion that DevOps is in reality a reaction to the
failure of Agile. Our basic premise was that the Agile founders believed that once
organizations implemented the upstream portion of Agile that over time they would as
169
part of natural progression develop the downstream portion of Agile. This natural
progression has simply never materialized and thus DevOps was born.
What we have also seen as DevOps was born was a lot of confusion between Agile and
DevOps inside organizations. This confusion is also causing a great deal of harm inside
many organizations, since these organizations are viewing Agile and DevOps as two
separate streams in their organization wide Agile transformation. The problem is that
many of these organizations are now creating two distinct silos in the name of Agile and
DevOps.
In order to put the new silo creation process in check, we need to investigate exactly
what the differences are between Agile and DevOps. Both Agile and DevOps are based
upon the same foundations of Lean and Deming. The only real differences between the
two is that DevOps looks to emphasize the breaking down of barriers between developers
and operations in order for them to collaborate and focus on the overall delivery of software
all the way to the customer and not just simply working software.
This focus on the delivery of software all the way to the customer differs from Agile’s
focus on Continuous Integration and Continuous Delivery, since DevOps also
incorporates Infrastructure as Code, configuration management, feedback loops, and
monitoring.
As you can see, DevOps is not a separate concept but a mere continuation of Agile to
include operations as well in the definition of cross-functional Agile team in order to focus
on delivering software fully to the customer.
Again, to emphasize, creating separate Agile and DevOps silos in any organization just
defeats the whole purpose of DevOps
170
It’s about Culture!!!
“Gartner Highlights Five Key Steps to Delivering an Agile I&O Culture.” Gartner, 20 Apr. 2015,
www.gartner.com/newsroom/id/3032517.
Ok, so we have pointed out numerous times in the book that DevOps is about culture
change and we have even delved into Organizational Behavior as a way of helping us
understand how to bring about change in an organization. What we have yet to discuss is
the specifics of what this culture change is that we are talking about in regard to DevOps
and the organization.
There are many areas of culture change that are required in DevOps and the fact is that
many of these areas of culture change are all too often mistakenly viewed as tool or process
issues instead of what they really entail and that is culture change. Below we will discuss
a few specifics of what these areas of culture change include.
One of the first areas of culture change that comes to mind is Unit Testing, which almost
universally is viewed in many organizations as being a tool, process or even developer skill
issue. This is a false assumption by most organizations, Unit Testing is very much a
cultural change issue. If we look closely at Unit Testing, we find that most organizations
already have most of the tooling in place to accomplish Unit Testing, many organizations
have brought in consultants/trainers to show their developers how to write Unit tests,
these organizations have probably had initiatives to start Unit testing and they have
probably planned for the inclusion of Unit Testing in their Agile iterations, so then why
are none of them doing Unit testing? The reason is that they have failed to understand
that adopting Unit Testing is a culture change issue and that unless they are up to
171
breaking down the cultural barriers surrounding Unit Testing they will never be
successful in implementing Unit Testing.
Another cultural change issue that is often misunderstood is the concept of incremental
releases. The concept of getting every developer to write small snippets of code and then
test the code and finally check it in immediately sounds like a fairly straight forward
concept, but we have to ask why can’t most organizations accomplish this? Again, we are
discussing a cultural change and not a tool or process issue. Over the years the culture
of software development has been to create branching strategies that are often long lived
and for developers to hold onto their code and only occasionally check code in. Breaking
this model that has been ingrained into the organizational processes and individual
developers means overcoming cultural barriers and not simply moving to a new source
code repository tool.
Another cultural change involves the whole realm of automated testing. For many years
the culture at most organizations has been for the Quality Assurance group to be
responsible for all aspects of quality and testing. With the emphasis on automated testing
first brought about by Agile and later required for DevOps, most organizations have gone
about implementing automated testing completely wrong due to their focus on tooling and
not the cultural issues that need to be resolved in order for it to be successful. So, what
are some of the cultural barriers that need to be changed in order for automated testing
to be successful? First, we have to understand that automated testing no longer falls
under the realm of Quality Assurance and instead is a piece of the process that is owned
by everyone. Organizations also need to understand that developers now need to take on
responsibilities for quality via Unit testing and other areas of automated testing in order
for automated testing to be successful. Lastly, it must be a conscious acknowledgement
by the organization that automated testing is coding and that means the individuals
creating the automated testing must have programming level skills and that learning
automated testing cannot be done in a two or three-day class.
There are many other areas of cultural change that are mistakenly viewed as being tool,
process or skill issues that organizations need to work to change in order to be successful
in the move to DevOps, in order to do this these organizations must first focus on
identifying the culture barriers to the various pieces of DevOps and only after that can
they focus on tools and adjusting the processes.
172
Dolphins Ah Ha Moment
The CEO is extremely excited about the presentation the partner firm will give since she
knows this will allow her and her management team to decide on the direction forward
and map out their strategy.
The consultants for the partner firm start out their presentation by explaining that they
will be covering DevOps in their presentation because they believe that Agile, CI and CD
should be looked at as components of what makes up DevOps. They state, that in this
day and age they believe that all companies must start the journey towards DevOps,
otherwise they put themselves at extreme risk of becoming uncompetitive and eventually
no longer viable in their market vertical.
The consultants next explain that DevOps is all about cultural change but are clear to
point out it’s not about culture change for the sake of change, but the culture must be
changed to support tried and true methods and processes that they will cover in the
upcoming slides. The first area of change they cover is the need for the organization to
start treating IT as a critical component of the business and not as a support department.
In short, they explain that IT must get out of the ticket taking mentality they have evolved
into and instead should be looked at as the core component of every company. The
consultants give some examples of this from other companies making the move towards
DevOps and how the executive management has proclaimed that IT is an indispensable
piece of what they do. Next the consultants detail the organizational structure most
companies have adopted and that in order to be successful they may need to change. The
three organizational structures highlighted are functional, matrix and market oriented.
Its explained that many of the successful DevOps movements have moved to a market
oriented organizational structure, but that some have been successful using a functional
organizational structure. The consultants are quick to point out that the companies who
are functionally organized and have been successful moving to DevOps have generally had
a culture in place that was already conductive to DevOps.
The consultants next discuss what DevOps is and use the Lean manufacturing
revolution of the 1980’s as a direct comparison. The Consultants explain what the Lean
Manufacturing revolution was, the outcomes it had and its relevance to the DevOps
movement. At this point one of the consultants starts explaining that much of this was
based on the teachings of Dr. Edwards Deming, this immediately arouses the attention of
the CEO. As the consultant explains how DevOps is based on much of Deming’s work and
others such as Dr. Goldratt, the CEO immediately realizes this is all in line with what she
173
learned during her MBA program. The consultants continue with their presentation of
DevOps, but at this point the CEO feels a sense of relief and excitement knowing that she
now knows the way forward. At the conclusion of the presentation, the CEO instructs the
CTO to not only hire the firm, but to set up a first steps meeting with the firm and that
she will be participating in the initial getting started meetings.
Breakdown of the Ah Ha Moment:

1.
2.
3.
4.
174
Learning Point(s):
Organizational Structures
Functional U-Form
175
Matrix Form
Marketing Oriented M-Form
In DevOps, organizations should be striving to create an organizational design that

effectively attempts to divide an organization into discrete parts (teams/groups) that are
given the required degree of ownership to make decisions in regard to the products in
176
their sphere of influence. In other words, we want decisions to be made by the most
appropriate people.
There are three approaches organizations may attempt:
Group resources together according to their functional purpose, examples are Quality
Assurance, Development and Operations.
Group resources together by the characteristics of the resources themselves, examples

are clustering skills together such as DBA’s, system admins or It may also be done
according to the resources required for particular products or services.
Group resources together by the product markets which the resources are intended to
serve.
Approaches to Organizational Design:
The U-form organization. A typical unitary form, or U-form organization silos its
resources primarily around functional purpose. This silo structure can unfortunately
favor process efficiency above customer service and the ability to adapt to changing
markets. The U-form organization has a tendency to build bureaucratic structures. The
silos inside these organizations often become primarily concerned with their own survival
and power. On the positive side, this type of organization keeps together expertise and
can promote the creation sharing of technical knowledge. The general issue with the U-
form organization problem then isn’t the development of capabilities, but the flexibility of
their deployment.
The M-form organization. The M-form organization came about because organizations
discovered that the U-form silos had become cumbersome as organizations expanded. The
M-form attempts to group resources around products and services. M-form organizations
often will distribute functions such as operations throughout the various product service
teams, this can sometimes lead to a reduction in economies of scale and operating
efficiency. M-form organizations are very effective at allowing individual teams to focus on
the specific needs of their market.
Matrix forms. The Matrix organization is a hybrid form, which usually combines the M-
form with the U-form. In a matrix organization resources are clustered around two
different lines of authority, often to a division level and silo group. The Matrix type of
177
organization attempts to ensure that all interests are represented, but they end up being
overly complex and confusing.
N-form organization. In an N-form (“N” stands for Network) organization, the resources
are bundled together into groups just like the other organization forms, but these groups
are given more delegation of responsibility for the strategic management of their resources.
The N-form organization tends to have little hierarchical reporting and control. Each of
the networked bundles of resources are linked to the others to form a network. In an N-
form organization the executive management works to set high level goals in order to build
a unified culture, but they do not attempt the traditional command and control type of
leadership as the other organization forms. As part of creating a unified culture,
management may work to encourage any development they see as beneficial to the
organization as a whole.
178
IT as Core Business Component
An issue that simply is affecting organization after organization who are attempting
Agile and DevOps is getting the business side to not only support the Agile and DevOps
movement financially and through resources but changing their mindset and
understanding that in this day and age the IT side is the business now and that the
traditional business side is really the support now. One of the lessons taught when looking
at the history of Lean Manufacturing is that IT in almost all organizations has in real
terms become the manufacturing floor of today’s organizations. If we were to look at an
example of this today, we could use a bank. In years past if you were to study how a bank
functioned you would have seen accountants who would calculate the balances of the
bank and customers via paper and pencil in ledgers. If a customer wanted a loan this
would have most likely required a financial analyst to calculate the interest and payment
via calculator. If one thought about using a credit card there would have been no way to
check if the user had acceptable credit and you wouldn’t have even thought about online
banking.
179
If we look at today’s world of banking, it would be unfathomable to even imagine a bank
operating without software to handle every phase of the banks business. You just have to
ask yourself this simple question, could you imagine any possible scenario where a bank
in today’s business environment could survive without a complex set of software
applications to run their business? The answer is no, and a definite no with the way that
some banks have been able to adopt Agile and DevOps.
This then begs the questions in almost every other business vertical, could the
organization survive without software. The answer is probably more difficult to answer
then you would think, because so many employees inside organizations are not truly that
aware of all the facets software is used inside their organization. This is seen in the
business side of most organizations today who are contemplating attempting Agile and
DevOps, the employees on the business side still view IT as some number they call
whenever their computer is not working properly. In order for Agile and DevOps to take
off in most organizations, the business side has to come to the conclusion that IT has now
become the core organizations business and that in order for the organizations to grow
and stay competitive, movements like Agile and DevOps must be viewed through strategic
visions and goals that must be properly funded, resourced and above all supported by the
business side.
IT is the new Factory Floor
180
One of the messages that should be repeated over and over by every Agile and DevOps
evangelist to every single organization that is contemplating Agile and DevOps is that
manufacturing has inspired most of the Agile and DevOps fundamentals. Organizations
must be taught that when we discuss the correlation between Lean Manufacturing and
DevOps, we are discussing the concepts, practices and methods used in Lean and how
that knowledge can be applied to DevOps. If you take a very close look at Lean
Manufacturing and its focus on automating work stations for example, you will never hear
those involved discussing which software to use in the automation of the work station.
What you will hear them discuss is what exact part of the work station process are they
going to automate, how will it affect work stations downstream and upstream (could it
cause a constraint), how will it cut cycle time, how will it culturally affect the employees
who may work at that work station among other things. Only after they have thoroughly
planned out the automation will they then discuss what software/tool to possibly use for
the automation. In comparison, if you go into most organizations looking to move to
DevOps the very first discussion they will have is whether they should use Puppet or Chef,
or in automated testing should they use Selenium or maybe Protractor. Lean
Manufacturing has laid the blueprint for using Lean concepts to improve productivity,
quality and speed to market, its inherent on those leading the DevOps movement in
organizations to heed this information.
Lean Manufacturing Concepts applicable to DevOps:
Standardized Parts - Following the lead of Lean Manufacturing, organizations when

putting together their tool kit for DevOps should look to tools that are standardized. When
we discus standardized parts an example could be a functional automation tool such as
Selenium which can be used across applications built with just about any programming
language. This means a team building an application built on Java can use the same
Selenium toolkit as another team building an application built on .Net. Organizations
should also look towards Open source tooling since it is more in line with the concept of
standardization versus a number of commercial tools. Another part of standardizing tools
is using tools from multi vendors and multi-platform. A number of tool vendors will
unfortunately sell organizations the notion that their tool kit if installed will solve all their
problems, unfortunately the truth is just the opposite. What you will discover if you buy
into a single platform from a vendor is that as you build out your infrastructure you will
back yourself into a corner.
Move to Modularization - If you take a look at Lean Manufacturing, one of the lessons
learned there is the reuse by the auto industry of automobile platforms. Almost all
181
automobile manufacturers reuse auto platforms across multi vehicle models, so if you’re
driving a Chevrolet Volt, it is built on the exact same platform as the Chevrolet Cruze. In
DevOps, the use of modularization involves micro services, which allow for software
component reuse. When using micros services, organizations should follow some basic
guidelines such as, a micro service should only constitute a single function, individual
micro services can have their own build, can be built using a number of languages, are
normally built by a single individual and have a stateless business logic with data caching
only when needed.
Automate all repeatable processes - This is an area where organizational Operations teams
truly need to focus. We obviously want to automate as many parts of the infrastructure
and pipeline process as possible, but many operations teams fail to look at their day to
day internal activities for possible automation. This is a problem because in many
organizations there is a shortage of operations resources to help with the infrastructure
and pipeline development, but if they would look to automate some of their typical daily
duties they may discover they free up additional resources. An example we have seen is
the setup of a new employee on the network that involved several manual steps that took
an operations resource almost thirty minutes to complete, when the operations team
finally investigated automating this they discovered they could automate the entire
process and as a result freed up several resource hours every month to work on
infrastructure and the delivery pipeline.
Standardize infrastructure - As we explained at the beginning of this section with our

example of a lean manufacturer looking to automate a work station, they would first do
all research and planning before they would even look into what software/tool to use.
What they have learned is that “processes drive tool selection, not the other way around”
Instrumentation & Control - One of the principles that should be borrowed from Lean
Manufacturing is their absolute focus on not only monitoring everything, but also
measuring everything. In DevOps, this involves telemetry and monitoring which we will
discuss in more detail in another section.
Continuous Improvement - The concept of continuous improvement is at the heart of the

entire Lean philosophy, we can see this thoroughly explained in concepts such as Kaizen,
PDCA and systems thinking. In DevOps, we accomplish this through feedback loops.
182
Dolphin puts Their Strategy in Place
At the first steps meeting the CEO, CTO, CFO and several other key managers are in
attendance. The consultants start out the meeting by explaining the artifacts that this
team will put together in order for Dolphin to get started on their DevOps journey. The
first artifact that is discussed is the need to create an organizational canvas to determine
the organizational structure that Dolphin will use to move forward with DevOps, the
consultants will discuss this in detail in an upcoming meeting and also provide additional
resources to help Dolphin make this decision. The next artifact that will be put together
is a DevOps road map that will help provide direction to everyone involved in the DevOps
implementation. The consultants explain that the road map will be used to make the very
difficult decisions that are going to have to be made in order for the DevOps movement to
have any chance of success. The consultants then give the Dolphin managers some
backing evidence showing that 90% of companies who attempt DevOps and do not deal
with many of the cultural changes contained in the road map have failed.
The last artifact that will need to be created is the Value Stream mappings of all of
Dolphin’s product lines. The consultants explain that Value Stream Maps is a tool used
to improve a process by identifying added value and eliminating waste. In other words,
it’s a process map that follows the value creation process. They then explain that this is
what will be used to determine where to start the DevOps implementation.
The consultants then proceed to discuss point by point a number of the difficult
decisions that are going to have to be made as part of the road map. The first decision
that will need to be made is what forms of the Agile methodology Dolphin going to use.
The consultants explain that there are a number of Agile methodologies and they aren’t a
one size fits all. The consultants will work with Dolphin to understand the various
methodologies and help them pick what will work best for them.
Next, the consultants move on to changing the incentive structure at Dolphin. The
consultants get into the Blame game and explain that most companies have actually put
into place disincentives to various departments working together. They explain that in
order to stop the blame game and break apart silos they need to change how they
incentivize their people. A critical learning point they then make is that almost all
successful DevOps movements have started from within and that dictating the DevOps
movement from top down has not been as successful. The consultants then explain what
this means is that you need to incentive your people to take charge of these new processes,
tools and ideas and run with them. In addition, if people are incentivized to try new ways,
183
then this will help break down the culture of fear that exists in most Organizations. By
changing the incentives, you give a sense of ownership to the employees, which in turn
helps foster a culture of learning.
The next area that will need to be tackled is training. The consultants make it very clear
that investing in training is not only a requirement for DevOps, but that it’s a smart
decision that will have huge returns in the long term. The consultants also explain that
the DevOps movement has seen companies move away from relying on contractors and
off/on shoring since there is a high investment involved in ramping up an organization
for DevOps. The consultants explain that most companies don’t want to absorb the cost
of training non-employees and then have them leave, in addition this also makes it difficult
to build a DevOps culture if resources are coming and going.
Another step in the roadmap is making sure the business side is brought along on the
DevOps journey. The consultants again emphasize how critical it is to get the business
side on board with DevOps, especially recognizing the profound changes DevOps has on
all areas of the business. The consultants explain that the business must be sold on how
DevOps will benefit them directly and not just shown a set of figures touting DevOps
benefits. They explain that it’s critical that the business understand DevOps and
participate. Since the business holds the purse strings, they will be needed to allocate
funds and resources to implement the infrastructure needed for DevOps, by determining
the value of a DevOps infrastructure request against their application requests.
The next area that is covered is the need to not only evangelize the DevOps work, but
also ensure that the progress is transparent, be it good or bad. The consultants explain
that when an organization attempts a change such as DevOps or even Agile they find that
the employees fall into three general categories, the innovators who see the value in the
change and will drive it forward, the laggards who will fight any change and sometimes
attempt to sabotage the change efforts and the majority of employees who fall into the
followers’ category and will follow whoever is deemed to be winning between the innovators
and laggards. Because the majority of the employees will follow, it’s important to
evangelize the result and progress in order to keep the laggards in check.
The last area the consultants bring up at the meeting is the need to put a plan in place
for dealing with the change in job responsibilities that come with an effort like DevOps.
The consultants specifically discuss how DevOps requires a change in mindset in regard
to testing. They discuss how DevOps requires what they term flipping the testing pyramid,
which means Dolphin will need to focus on automated testing at all layers and also shifting
184
the testing left so that its done earlier in the process. The reality of this change in testing
means that new skills will be required along with a new mindset in how testing is thought
about. The industry research has found that in almost no instances has this shift in
testing been successful when undertaken by traditional quality assurance departments.
The consultants explain that first, these traditional testers normally lack the technical
skills required to do automated testing and more importantly when the developers are not
involved the necessary feedback loop to the developers is never created causing the
automation efforts to fail.
To wrap up the meeting, the consultants suggest that it would greatly enhance the
chances of success if the managers would read the two DevOps books they have
suggested. They explain that the managers will be called on to support the DevOps
movement and that in order to do that they need to have a grasp of what DevOps entails
so that they can provide the necessary support. After the meeting, the CEO sends out an
email to all managers requesting that they read the two DevOps books and in addition
asks her CTO and CFO to engage their subordinate managers in discussions about what
they learn in the books.
Breakdown of the Strategy:

1.
2.
3.
4.
185
Learning Point(s):
Value Stream Mapping
When any organization decides to take on the challenge of moving to DevOps, one of
the very first orders of business for the DevOps leaders should be to hold a Value Stream
Mapping exercise. One of the reasons why we want to start with a Value Stream Mapping
exercise is to discover where we will start our DevOps implementation. Most organizations
produce more than one software application, so we need a way to help us determine which
application (pilot application) we will start with before attempting to scale across all the
applications. It is generally not advisable to attempt to scale DevOps across all
applications at once, due to the fact that there is no such thing as DevOps in a box, which
means there will be trial and error during the initial pilot project.
186
So, what exactly is Value Stream Mapping? Value stream mapping is a lean-
management method for analyzing the current state and designing a future state for the
series of events that take a product or service from its beginning through to the customer.
Value Stream Mapping, has also been called other names, such as at Toyota, where it is
known as "material and information flow mapping”. Value Stream Mapping can be applied
to nearly any value chain (process).
Generally, value stream mapping has most often been associated with manufacturing,
but it is also used in logistics, supply chain, service related industries, healthcare, product
development, and administrative and office processes.
Why Value Stream Mapping is Important
Understanding your internal value streams is critical to both to serving the needs of the
customer, and to improving the efficiency (and profitability) of the value stream. Value
stream mapping is a means of representing the value stream with symbols and numbers,
which is the key to understanding the entirety of the transformation of ideas into finished
products. Therefore, when we study the process, it is therefore also essential to consider
the role of information in the internal value stream. Information comes, ultimately, from
the customers; their requirements are translated into specifications, code, and delivery
functions.
Value Stream Maps are not Process Flow Diagrams
Traditional process flow diagrams simply do not tell the whole story, process flow
diagrams, whether drawn by hand, or created using specialized software lack a critical
element. Traditional process flow diagrams only present a “snapshot” of the sequence of
steps in the flow. We need to think of process flow diagrams as pictures of actors and
scenery, while a value stream map would be described as a movie. Process flow diagrams
thus lack the element of time. Time is essential to understanding how one operation affects
another, how a particular resource may influence the entire process, and how the status
of the queues and operations vary with time.
Using Value Stream Mapping in software development
As the DevOps movement has realized the overall success of Lean in manufacturing
and production, this has in turn led to an interest in its adoption in software development.
187
We must be cautious here and realize that many individuals and vendors have not fully
understood that much of the current literature on adoption of Lean in software
development has had a disconnect between the high-level principles and the concrete
practices related to lean. Much of the current literature in regard to Agile and DevOps has
had a limited focus on wastes that were literally mapped from the categories identified for
manufacturing. This literature has ignored the transformation that lean thinking has itself
undergone and moved away from the focus on "removal of waste" to "creating and
delivering value".
Value Stream Metrics
The Two key metrics associated with value stream mapping is value adding times and
non-value adding times. Non-value adding time is called waste or muda.
Murphy’s Curve
Slack, Nigel, et al. Operations Management. Pearson, 2017.
188
Leading a DevOps implementation is a challenging proposition for a number of reasons.
One of these reasons is not only getting the business side to buy in, but to keep them on
board throughout the entire journey. The truth is that many organizations, vendors and
coaches/consultants don’t fully appreciate just how steep the mountain is that must be
climbed in order to bring about the culture change required to successfully implement
DevOps.
The implementation of DevOps will need to account for the ‘adjustment’ issues that
almost always occur when making any organizational change. By adjustment issues we
mean the loss in performance that could be incurred before Agile and DevOps are fully
implemented and functioning as intended. But estimating the nature and extent of any
implementation issues is notoriously difficult.
This is particularly true because more often than not, the difficulty in large scale change
(cultural) is greatly under calculated. When discussing Agile and DevOps related change,
we can attempt to use Bruce Chew of Massachusetts Institute of Technology’s ‘Murphy
curve’ argument that adjustment ‘costs’ stem from unforeseen mismatches between the
new technology’s capabilities and needs and the existing operation. In Agile and DevOps,
this would be unforeseen mismatches between Agile and DevOps cultural needs and the
organization's existing culture.
Implementing Agile and DevOps changes, processes and tools rarely behave as planned
and as changes are made their impact ripples throughout the organization. Above is an
example of what Chew calls a ‘Murphy curve’ applied to DevOps. It shows a typical pattern
of performance reduction as DevOps is introduced. It is recognized that implementation
may take some time; therefore, allowances are made for the length and cost of a ‘ramp-
up’ period.
However, as the operation prepares for the DevOps scaling, the distraction causes
performance to actually deteriorate. Even after the start of the scaling this downward
trend continues, and it is only weeks, indeed maybe months, later that the old
performance level is reached. The area of the dip indicates the magnitude of the
adjustment costs, and therefore the level of vulnerability faced by the organization.
189
Single versus Double Loop Learning
Understand that improvement is learning.
It should not be a surprise that Agile and DevOps improvement implies some kind of
intervention or change to the process, and the change will be evaluated in terms of
whatever improvement occurs. The evaluation that takes place adds to the knowledge of
how the process really works, which in turn increases the chances that future
interventions will also result in improvement. What is critical to remember is that it is a
learning process, and it is crucial that improvement is arranged so that it encourages,
facilitates and exploits the learning that occurs during improvement. We must as a result
recognize that there is a distinction between single- and double-loop learning.
Single and double loop learning
Single Loop learning occurs when there is a repetitive and predictable link between cause
and effect. Quality Assurance, for example, measures output characteristics from the
development process, such as defects, adherence to requirements/acceptance criteria,
etc. These can then be used to alter input conditions, such as user story/acceptance
criteria quality, standards compliance, developer skill, with the intention of improving the
output. Every time a development error or problem is detected, it is corrected or solved,
and more is learned about the process. However, this happens without questioning or
190
altering the underlying values and objectives of the process, which may, over time, create
an unquestioning inertia that prevents it from adapting to a changing environment.
Double Loop learning, on the other hand, questions the bottom line objectives, service
or even the underlying culture of the process. This type of learning implies an ability to
challenge existing process expectations in a fundamental way. It seeks to re-frame
competitive assumptions and remain open to any changes in the competitive environment.
But being receptive to new opportunities sometimes requires abandoning existing process
routines which may be difficult to achieve in practice, especially as many processes reward
experience and past achievement, instead of potential at both the individual and a group
level.
Theory of Constraints and Five Focusing Steps
In this section, we will explore why the Theory of Constraints and the Five Focusing
Steps when used in DevOps can be so critical to the bottom line of most organizations. In
DevOps, our whole focus is on converting all of the individual process scattered about the
organization into a single process or pipeline. The Theory of Constraints is completely in
191
line with this thinking because it has a holistic view, which only focuses on the big picture,
rather than the small details. TOC reinforces that instead of managing separate resources,
assets, or procedures to optimize their productivity, the focus should be on the
interconnections and links among these separate components, specifically on the barriers
that impede them from working as a single unit. When TOC is correctly used the result is
an entire system that flows and operates smoothly, free of any bottleneck.
Benefits of using the Theory of Constraints
Reduction in Inventory - In the case of software development this means a reduction in

work in progress (WIP) and our Agile backlogs. By improving processes through the
management and elimination of bottlenecks we can sharply reduce cycle times in the
development process, which means we have less work waiting in the pipeline and the Agile
teams in many cases can increase their velocity reducing our backlog.
Reduced operational costs - There are countless ways that the elimination of constraints
can reduce costs of operations. One obvious reduction in operating costs is the reduction
in manual testing cycles required when a pipeline becomes fully automated.
Enhanced throughput - As we have discussed in an earlier section, the benefits of using

TOC in DevOps to eliminate constraints, thus allowing us to automate the delivery pipeline
has shown on average to reduce cycle time by ninety percent.
Improved workplace synergy - As discussed in the Blame Game section, by eliminating

constraints that are blocking development and operations from working together we can
not only improve workplace synergy, but can allow for more time spent on actual
productive duties.
Better operational control - The TOC provides this guidance to teams, so that they can
focus on what truly needs to be prioritized.
Improved reputation - Using TOC in DevOps results in improved processes, which means
that products and services will be delivered to customers on time, in full, and with no
defects, thereby improving the company’s relationship with the customers.
192
Evangelize, Evangelize and keep doing it
An interesting DevOps experiment would be to select a handful of organizations that

have failed in their attempt at DevOps and a handful who have succeeded and visit each
organization without knowing whether they were successful or not and see if there are
any outward signs that could be observed in the first 30 minutes that might suggest
success or failure in their DevOps attempt. There is a high probability that there would
be outward signs that identify who was successful and who failed. The reason for this is
that the organizations who were successful in their DevOps attempts most likely were
affective in evangelizing the DevOps movement inside the organizations.
There are any ways in which the DevOps leaders can go about evangelizing the
movement and a lot of very good practical examples in various DevOps books. There really
is no one way of evangelizing and there are also many forms of media to use in order to
get your message out, instead we will discuss several factors to help ensure DevOps
message being sent is successful.
Take Ownership and Overcome Fear
193
In order to lead the DevOps movement inside your organization you must have a sense
of self-awareness and be willing to take ownership The DevOps leaders must be willing to
engage their ideas and qualify them for further actions, this often involves going beyond
the traditional set of options inside the organization. The leaders of the DevOps movement
must not only have a deep conviction for why DevOps is needed, but they must be able to
support this belief with facts. It is also important to be able to explain the current state of
affairs (Value Stream Mapping) and where you want to take the process. Too articulatory
persuade the DevOps benefits, it has to be connected to the DevOps leader’s own
convictions. Simply put, the DevOps leaders must overcome any fears of evangelizing their
ideas.
Disconnect Personal Gains from Organizational Objectives
The DevOps leaders in order to be credible must be able to show that the benefits of
DevOps are disconnected from their own self benefits and aspirations. In order for DevOps
to be a change agent the DevOps leaders will need to be able to articulate how the
modification of processes will directly benefit the organizations goals and not their own
personal gains. This means the DevOps leaders need to understand the boundaries
between their own desires and the changed outcome brought about by DevOps.
Must Earn Trust
It is critical that the DevOps leaders be able to earn the trust of those who will be
affected by DevOps. The only way to build this trust is to be able to demonstrate
competency in their leadership, demonstrate a trust worthy character and show an
absolute resolve to the DevOps cause. It’s always important to understand that trust is
earned and never simply awarded.
Competency
As has been mentioned throughout the book, in order to lead the DevOps movement
inside an organization, the DevOps leaders must take the initiative to learn the
foundations of DevOps and core components of DevOps. DevOps is about continual
learning and improvement; how can one lead this type of movement if they in fact are
unwilling to continuously learn and improve?
Those without Character Will Fail
194
History has shown that in order to aspire change in others, one must themselves be
able to demonstrate a level of character that inspires others. Those with high character
and principles consistently reflect this in the work they do, and others observe.
Remember, it’s not just the work you have done, but it’s also about the attitude you have
that others see.
Be Committed to the Cause
All too often in organizations the employees have seen management trot out new ideas,
concepts and methodologies and then watched as management lacked any real
commitment and the efforts simply faded away. In these organizations, there is already a
built-in belief that any new ideas will be seen as just talk and that the organization will
never commit to the cause. The DevOps leaders must have an unflinching commitment to
DevOps even when they experience failure and stumbling blocks.
Build Relationships
To get others to cooperate, the DevOps leaders must set out to engage others who can
help, build a network of relationships in the organization and above all show empathy to
those they are trying to change. The DevOps leaders must understand that in order to
persuade others, they must have the ability to empathize with others.
Educate
The DevOps Leaders during their evangelizing should attempt to create an atmosphere
of comfort, start small, and above all educate the organization members. The education
should include management, but this education should not be perceived as threatening
their perceived sense of power. Research has shown that one influential practice of
educating various parts of an organization can be done effectively through the use of
storytelling. Storytelling has been shown to be an effective way of transferring knowledge
to help people see the reason for change.
Understand Other’s Perspectives
Change only happens when others feel they are being equitably rewarded by the change.
The DevOps leaders need to package DevOps in a way that addresses the needs of
individuals and groups in the organization, takes into account their motivational values,
and their philosophies. There should never be an attempt to manipulate, instead true
195
leaders motivate. The DevOps leaders need to focus their intent on the underlying
inspiration of the DevOps evangelizing effort so that it’s focused on others interests rather
than their own.
Prepare for Confrontations
The DevOps leaders must prepare for when individuals/groups attempt to change or
undermine the DevOps efforts. The plan for confronting these individuals/groups should
be based on factually describing what happened and what was expected. When
confronting the violators, they should be given the chance to redeem themselves.
Don’t Give Up
DevOps is not something that can be implemented overnight and it will also be full of
setbacks, but in order to be successful the DevOps leaders must be persistent in their
efforts to bring about change.
196
The Difficult Choices Dolphin Initiates
The management team at Dolphin reads the DevOps books suggested over the next
couple weeks and finds that it ignites much discussion among the managers. A number
of managers comment that the light bulb has gone on for them regarding how they have
managed their departments, interacted with other departments and understood how
processes should work. During this time, the consulting firm and the management team
selected by the CEO also continue their meetings, the consultants continue working with
the managers to iron out some of the decisions that must be made by the executive
management team.
The CEO now believes her team has consensus on the decisions her team must make
in order for the DevOps movement to get started within the organization. The management
team starts by changing the way the entire incentive/pay structure works at Dolphin.
Starting immediately the employees will have their entire incentive and promotional status
based on whether or not they are seen as not only either trying or improving processes
and tools, but if they are doing it in a way that facilitates global learning by the company.
Simply put, the CEO makes the point that employees will not try new ways of doing things
unless they are encouraged to do so and that they do not have to fear the possibility of
failure.
The next change that the management team tackles is the organizational structure of
Dolphin. After much consideration, the team decides that Dolphin will move from a
functional organization to a marketing organizational structure. This decision is made for
a couple of reasons, first in order to encourage the idea that IT is a core component of the
company and its strategies, they need to start breaking down silos. In addition, the team
also has looked at the Operations department and determined that with the current
resources available and the current structure, it would be advisable to move these
employees onto Agile Platform teams moving forward. Lastly, the team feels that to
encourage ownership and take advantage of the feedback loops that will be built as part
of the DevOps implementation, the teams should be organized around the products
themselves.
The next decision the management team tackles before handing off the DevOps to the
organization is the decision of how to handle the Quality Assurance team. After much
discussion and deliberation an answer has been reached with the help of the Quality
Assurance managers. The Quality Assurance managers after getting assurances that the
team will not be let go, has proposed a solution that they believe aligns with not only what
197
Quality Assurance is supposed to be, but also the teaching of Lean and Deming. The
Quality Assurance accepts that they have not actually been doing Quality Assurance work
for several years, instead they have been providing Quality Control by testing after the
fact. In order to align with true Quality Assurance which is the idea of continually
improving processes, the Quality Assurance team will be moved to other areas such as
providing testing and product knowledge to the members of the Agile teams, working with
the DevOps teams to foster Quality Assurance and they will also be offered the opportunity
to receive the training necessary in case they wish to move to automated testing.
The last decision the management team makes is that members of sales, marketing and
HR will be incorporated into the DevOps Agile teams to help foster the necessary
organizational learning that must occur for DevOps to be successful. With these decisions
made, the management team hands off the DevOps scaling to the organization, but with
assurance they will be monitoring and providing support and possible direction when
needed.
Breakdown of Dolphin’s Choices

1.
2.
3.
4.
198
Learning Point(s):
Everyone is Responsible for Quality
One of the foundations of the Agile methodology is that everyone is responsible for
quality now, yet this is not the reality in the clear majority of organizations who have
attempted Agile or DevOps. All one has to do is look at the Agile practice in most
organizations and what one see is a Quality Assurance group who is still responsible for
all testing and owns all forms of testing in the organization. All too often you see Agile
teams who will have an iteration dedicated to coding the new change/feature and then
there will be a follow up iteration dedicated to testing the previous iterations code. This is
a clear indication that the organization has not been able to culturally change from the
traditional view of Quality Assurance as the gatekeeper of quality to a view of everyone
pitching in to ensure the product meets the requirements and specifications.
In the move to DevOps, the move from Quality Assurance being the gatekeepers of
quality is not a pie in the sky principle, but an absolute requirement. All too often Quality
Assurance is fighting the change from being the quality gatekeepers, because the industry
199
has forgotten what the actual role of Quality Assurance is supposed to be and because
Quality Assurance has failed to keep up with the changing landscape of the industry.
Quality Assurance should be leading the charge in DevOps due to the synergies between
the two. Quality Assurance is supposed to differ from Quality Control’s focus on defect
detection and rejection, since Quality Assurance is supposed to focus on defect prevention
by focusing on quality earlier in the process which have been referred to as a shift left.
What has happened in the industry is that Quality Assurance has been viewed as Quality
Control in most organizations and has little to no focus on improving processes in order
to improve quality.
The other issue is with Quality Assurance not keeping up with the changing landscape
of the industry. The industry starting with Agile and now being reinforced in DevOps has
moved to automated testing as a way of shifting quality left in the process in a continuous
manner. The move to automated testing should have been welcomed by Quality Assurance
since it focuses on detecting defects earlier in the process, which is what Quality
Assurance is supposed to be focused on. Unfortunately to many in the Quality Assurance
field have lost track of what the field is supposed to be focused on and instead believe they
serve as a Quality Control function.
Until organizations are willing to tackle the cultural change required to make everyone
responsible for quality and Quality Assurance catches up with the industry change to
automated testing and is willing to relinquish their hold on testing, the results will be that
automated testing in most organizations will remain a pipe dream.
200
Are you Siloed?
Traditional software development has seen the arrangement of teams around specific
roles, which has led to the formation of silos. The formation of silos inevitably leads to
poor communication and weak cross-departmental support. The silos created cause
problems for a number of reasons, first the employees in the silos become almost
exclusively focused on the goals of the silo rather than on the needs of the organization.
This can in turn cause important project priorities to be ignored in favor of the priorities
of the silo.
How do I Know if my Organization has Silos?
Detecting silos inside the organization is actually pretty easy to spot. If you commonly
observe a lack of collaboration among individuals and teams this is one, clear sign. Do
you see little innovation occurring in the organization and almost constant firefighting?
Another common theme of silos is the creation of unnecessary duplicated work, and lastly
201
do your Agile teams seem to have difficulty getting on the same page? Is there lots of finger
pointing when things go wrong and does it take a long time to get issues corrected? These
are all common signs of silos in the organization.
How DevOps Helps Break Silos
DevOps changes the organization on a cultural level, in DevOps it means that everyone
has to own the overall business goals. When DevOps is successfully implemented everyone
in the organization has to do their best to realize business value. The DevOps Culture
helps break down common barriers between teams such as lack of collaboration and
fragmented workflows. DevOps borrows the concept of cross functional teams, which
results in teams working together to support shared organizational goals and results in
better product quality.
Tackling Training
Throughout the book we have referenced that training is critical inside an organization,
but just how do you go about developing a training program for DevOps on the scale
needed without breaking the bank?
202
First though, let’s be crystal clear about what the DevOps training should be focused
on and who should be included. Any DevOps training program should concentrate on
developing an Agile/DevOps mindset, with the appropriate level of technical training
required. The training program should not be centered on technical/tool training. Any
training program that is developed should include all levels of the organization, that
includes the executive level. Why you ask should all levels be included? Think of it like
rowing a boat, if the rowers on one side are rowing the boat and the rowers on the other
side are not, what happens? The boat simply goes in circles, much like what is happening
with most Agile and DevOps efforts in organizations.
So, how can an organization go about developing an Agile and DevOps training program
that is scalable across the organization? One possible way is to borrow from a technique
used in Higher Education. This concept uses the idea of Cohort groups. The idea is that
small Cohort Groups would be created at all levels, preferably cross functional cohort
groups. Then using an appropriate tool, the groups would be given weekly assignments
(readings or activities) where they would be expected to discuss the reading or assignment
in their Cohort Group and then depending on the reading or assignment a chat tool could
be used to allow all the Cohort Groups to discuss the reading or assignment to further
knowledge transfer. Some Cohort Groups may have assignments such as Hackathons
where they would learn to use various DevOps tooling, possibly in competitions against
other Cohort Groups.
Orca and Dolphin’s Project
Both companies have been tasked with creating a new online membership rewards web
site that their customers can use to track their rewards status. In many cases, both
companies may be fighting for the same customer. The goal is for each company to get
their new membership rewards web site launched to existing and potential customers as
fast as possible, while also infusing quality into the product and meeting the customer’s
needs. The fate of both companies is riding on this project.
203
Let’s Examine Dolphin’s Strategy in Depth
Iteration 1
Company Dolphin – Uses Scrum in two week iterations along with
Specification by Example.
A. Company Epic creates four User Stories and corresponding Specification by
Example scenarios to initially develop a new Membership Rewards Login Page and
places them in the backlog.
Story 1 and 2 in the first iteration.
C. The team gets together (Three Amigos Session) and flushes out all of the
Specification by Example scenarios for the two User Stories that were taken in
during this iteration
D. The developers have a design meeting to flush out the details
E. The developers start building out the code in a TDD fashion using the Specification
by Example to drive the test design.
F. The developers build the code using monitoring driven development so that the new
code will have telemetry embedded
G. Another team member starts writing out the Functional automated tests in a TDD
fashion.
H. After the third day, developers start submitting Pull requests for code reviews.
I. The Continuous Integration automatically starts picking up the Pull requests after
code review sign off and builds, deploys and tests the code. Once the code has
successfully passed the Unit Tests, Functional Tests and Quality Gates it is
versioned and baselined and automatically pulled into the Acceptance phase.
J. The Product Owner accepts User Story 1 and 2 during the Sprint Review.
K. Once the code successfully navigates the Acceptance phase and the Product Owner
gives the green light, it is moved into Production with the Feature Toggles turned
“off”.
L. Move to Production Off section
204
1.
2.
3.
4.
Learning Point(s):
205
Telemetry and Monitoring
Telemetry
To fully put our feedback loops in place, we need to take a close look at the subjects of
Telemetry and Monitoring.
What is Telemetry?
The universal definition of Telemetry is the recording and transmission of data via an
automatic form from some remote or inaccessible source to an IT system in a different
location to monitor and analyze the data. The data from Telemetry can be sent using radio,
infrared, ultrasonic, GSM, satellite or cable, depending on the. The reason that Telemetry
is so critical in DevOps is that can provide insight into which feature the customer uses
most and it sometimes can detect bugs and issues before they are detected by the
customer. In other words, we can solicit feedback without contacting the customer directly
and in many cases without the customer even knowing.
Telemetry isn’t Logging
While logging is a highly recommended practice, we need to understand that it is

normally a tool we use during development to diagnose errors and code issues. General
206
Logging is typically used internally rather than with actual customers. Telemetry on the
other hand is used to collect data from actual customer usage.
Benefits of Telemetry
The key benefit of adding Telemetry to your applications is that it gives teams the ability
to monitor the state of objects while not physically present. The data that is sent from
Telemetry can be gathered and delivered into a dashboard for the team to analyze and
respond as needed.
What you can see:
Are your customers using the feature and how are they using it?
The frequency and duration of usage by the user.
Which settings are being used most often by the users.
Investigate the context of crashes.
What exactly happened during a crash.
Monitoring
In the DevOps market, today there are numerous vendors trying to promote monitoring,
choosing among them can be distracting. To sort through the various vendors and
determine which tool(s) are best for you, it’s advisable to sit down and determine which
types of monitoring hold the most value to you and your customers.
Understanding the Background of Monitoring
It’s important to recognize that various types of monitoring have been around for many
years and isn’t some new concept brought about by DevOps. Even though monitoring
predates DevOps, it is having an effect on monitoring in the sense that it has advanced
the software development process so much that it in turn is helping evolve monitoring.
What are we Monitoring?
In DevOps, we are monitoring a number of crucial categories. The first category is the
log output of our applications, here we are looking for errors and exceptions in real time.
The next category is the health of the servers, here we are looking at the uptime and
performance in respect to the available resources. Another category is milestones in the
207
development process, here we are really looking at the success of the DevOps
implementation.
Another critical category is the known vulnerabilities or weaknesses in our application

code and insecure coding practices. The next category and one of the easiest to setup is
failed builds and deployments. The last category is user activity monitoring, we use this
to help teams drive feature development and the scaling of our infrastructure.
Why TDD
Earlier in the book we discussed the need for Unit testing and that we wanted to
separate the concept of Unit testing from Test Driven Development (TDD). TDD is one type
of Unit testing, just like Scum is one type of the Agile methodology. In this section, we will
discuss why we believe TDD is not only in line with the teachings of DevOps but is a
208
critical piece of accomplishing some of the key components of DevOps. Below we will
highlight how TDD actually works hand in hand with DevOps.
Feedback Loops
While Unit testing does create a feedback loop for the developer and team who creates
the Unit tests, TDD actually creates a feedback loop earlier in the cycle, which helps to
prevent defects and improves quality. How does TDD create an earlier feedback loop than
traditional Unit Testing? In traditional Unit testing the developer would write the code and
only after the code was complete would the developer write the Unit tests and run them
to confirm the code works, thus only getting feedback after the fact. In TDD, the developer
would write a test that meets the requirements (preferably using Specification by Example)
and then would write the code to pass the test. This creates a feedback loop immediately
as to whether or not the developer is writing code that meets the requirements.
Higher Acceptance Rate
Because Specification by Example (also known as BDD and ATDD) has become almost
a subset of TDD, when paired with a DevOps environment, organizations will find that the
implementation of the code is more likely to match the product owners vision for the
requirements. Writing TDD test cases that are based off Specification by Example allows
the test cases to be easily generated from the acceptance criteria without interference from
the constraints of the architecture design or programming constructs. By pairing TDD
with Specification by example we guarantee to some degree that the final version will fulfill
the needs of the customer as communicated by the product owner.
Eliminate Waste
The combination of Specification by Example and TDD helps to prevents unneeded

design ideas or components to sneak into the product. Specification by Example guides
the test cases, to help define the exact set of required features. By using TDD along with
tools such as SonarQube, we make it easy to identify redundant code.
Customer Focused
TDD along with Specification by Example work well with a customer-centric agile
process that is at the heart of DevOps. Through TDD, our iterations can now be defined
209
as the implementation of functionality to execute against a pre-defined set of specification
test cases instead of the more traditional set of specifications.
Encapsulation
One of the requirements for DevOps is that developers write code in small snippets and
constantly check this code in. In other words, we are attempting to create code that is
encapsulated and independent. Through the use of TDD, we can create encapsulated,
flexible, and extensible code. The TDD technique requires that developers think of
software as small units to be written and tested independently and then integrated
together later.
Rev Up Quality with Code Reviews
Code reviews are a subject that unfortunately brings about too much debate among
programmers because they often are unsure how to perform them correctly and where
they belong in the development process. It is possible that you may need to make some
minor changes to your development workflow, and the review should happen before the
code is checked into the mainline.
210
Most organizations moving to DevOps have discovered that code reviews do seem to
work better when using a distributed version-control system, such as Git, which allows a
developer to work on their own local version of the repository. Once the developer finishes
their work, a simple pull request is made, which also contains a request for someone to
review the new code.
One concept that many organizations fail to understand is that while code reviews will
hopefully help find bugs in the code, they are not a replacement for a high level of
automated test coverage. Remember, the code reviewers should never be doing regression
testing in a sense, that is why we have automated tests. An absolute requirement before
any code review is performed is that the new code has sufficient automated test coverage
(Unit, Functional etc..) in place and all the tests are passing. The reason we still do code
reviews though is because sometimes the test cases are poorly written or have missed
some business logic, this is where the code reviews are effective. As a side note here, to
confirm the Unit tests are written properly and cover the business logic it’s not a bad idea
to run mutation testing on the Unit tests.
How to make Code Reviews More Effective
Its highly advisable and a DevOps best practice to create small branches since we
should be releasing the code in an iterative fashion. This makes the job of the code
reviewer much more manageable.
Hold reviews often
Since we should be checking in code in small iterations, we should be requesting code

reviews frequently. The job of a code reviewer shouldn’t take hours to perform, we do this
by keeping the task very small. Additionally, this allows the coder to receive feedback in a
timely manner.
Architect and Engineers
It is preferable to have the software architect and at least one other engineer perform
the code review. The architect should play the role of enforcing the architecture best
practices.
Should you Use a Tool?
211
Code review tools such as those in GitHub allow for code review to be performed much
faster and possibly more importantly, are part of the delivery pipeline. Tools like GitHub
also allow the participants to hold conversations and verify suggested changes are made.
Maximize Post Mortems
Many organizations are familiar with the basic concept of Post Mortems from either
their experience with Agile or from holding Post Mortems for production incidents. What
these organizations may not be familiar with is holding Post Mortems that maximize their
benefits.
There are two areas in which many organizations are failing to gain the full potential of
Post Mortems, we will briefly discuss both in this section. Many individuals and teams are
probably familiar with the basic setup of a Post Mortem, but they are probably missing
out on one of the areas where they can be of tremendous help. Most Post Mortems are
marked by individuals spouting out all of the issues that occurred during a previous
iteration and then the team marking what they need to improve. This is all fine and should
be brought up, but what about what has gone right? Identifying what has gone right and
212
putting as much emphasis on this as what has gone wrong or needs to be improved can
pay even bigger dividends in the long run.
The other area we need to discuss when talking about Post Mortems, especially when
they concern production incidents is making sure there is absolutely no downward
pressure by the organization on individuals/teams to not bring up issues such as security,
ethics and safety. If there is downward pressure to not point out these types of issues,
what the organization may discover is that one of the production incidents in the future
may be an organizational killer, because individuals/teams were pressured to not discuss
issues they were aware of.
Iteration 2
Company Dolphin
A. The team determines during the sprint planning session that they can take in User
Story 3 and 4 in the second iteration.
B. The team gets together (Three Amigos Session) and flushes out all of the
Specification by Example scenarios for the two User Stories that were taken in
during this iteration
C. The developers have a design meeting to flush out the details
D. The developer start building out the code in a TDD fashion using the Specification
E. The developers build the code using monitoring driven development so that the new
code will have telemetry embedded.
F. Another team member starts writing out the Functional automated tests in a TDD
fashion.
G. After the third day, developers start submitting Pull requests for code reviews.
H. The Continuous Integration automatically starts picking up the Pull requests after
I. During one of the Pull Request Builds, the Unit Tests catch an issue that breaks
existing code. The team swarms together to fix the pipeline. The Pull Request is
pulled back, fixed and code reviewed.
213
J. The Product Owner accepts User Story 3 and 4 during the Sprint Review.
“off”.
L. At the end of the second Iteration, the Login functionality is pulled from the artifact
repository and loaded into a UAT environment and turned on so that internal
customers can try the new functionality.
M. The team now watches the telemetry that is providing a feedback loop from the UAT
environment and talking with the UAT internal customers.
N. Enhancements are suggested, and the team is quickly able to make the changes
and get them through the Continuous Integration and Acceptance phases in order
to get them into the UAT environment
O. Move to Production Off section
1.
2.
3.
4.
214
Learning Point(s):
True Automated Testing
We will now discuss one of the most misunderstood subjects inside most organizations
today and that is the subject of automated testing. There are so many misconceptions
regarding automated testing, especially regarding the types of automated testing to do and
what to use. Below we will discuss many of the myths and misinformation regarding
automated testing.
The first bit of misinformation we will discuss is the individuals and vendors doing tool
comparisons of functional automation tools. If you come across anyone doing a tool
comparison between say a commercial functional automation tool and either Selenium or
Protractor, our best advice is to close the site and disregard anything you may have read.
You are probably asking why we suggest this advice, the reason is that they are comparing
apples and oranges. The commercial functional automation tools are just that, a set of
tools for functional automation testing. Selenium and Protractor are API’s and nothing
more, the commercial tools often come with their own IDE, test repository, ability to run
215
the tests and a limited number of languages to write the tests in. On the other hand, an
API like Selenium can be written in almost any programming language and uses the
languages corresponding xUnit framework tool to run the tests and report the results. You
may be asking yourself, well wouldn’t we want all these tools the commercial vendor offers,
the answer is probably not. The reason is fairly straight forward, if you are for example a
Java shop, wouldn’t you want to be able to use the same IDE you write your application
code to also write the automated tests, and wouldn’t you want to be able to write the tests
in Java if that is what most of your resources are skilled in? There are a fairly large set of
other reasons you probably would not want to go with the commercial vendors if you are
moving to DevOps, we will highlight a few more below.
Another mistaken misconception comes from users of many of these commercial tools.
When discussing an API like Selenium and the use of Page Objects you will often hear
them state that their tool is superior because it can generate their version of Page Objects
for them and Selenium users have to manually create their Page Objects. The problem is
that these commercial users are completely blind to the move to start writing even
functional automated tests in a Test-Driven Development manner. Using Page Objects
with Selenium and especially when combined with Specification by Example allows the
test writers to basically stub out the Page Object and tests in the same manner as a Unit
test and then code to make them run. Many of the commercial tools require the code to
be written before they can generate their version of Page Objects, making any attempt at
Test Driven Development futile.
A couple of other myths are that organizations can employee a record and play tool for
automated functional testing and achieve DevOps and that traditional Quality Assurance
can take a quick two or three day class and become full blown automated testing
engineers. The first myth regarding record and play, you will never in any situation be
able to accomplish DevOps with a record and play tool. The maintenance cost alone will
make this completely impractical. The other myth that traditional Quality Assurance can
take a single class and suddenly become automation engineers is simply not reality for
the majority of Quality Assurance. Make no mistake, in order to do automated testing,
one must possess programming level skills to do it properly.
The last misconception made by many organizations who are moving towards DevOps
is that they simply need to incorporate Unit Testing and some Functional Automated
Testing and they are ready for DevOps. Organizations who do DevOps actually employee
automated Unit testing and possibly Mutation Testing on top of that to test the Unit tests,
functional automated testing, automated security testing, backend and frontend
216
performance testing, web consistency testing, automated accessibility testing and
automated infrastructure testing. Most of the automated testing required can be found in
traditional Agile/DevOps testing pyramid.
Pipeline Swarming
A fairy simple but important DevOps technique that should be briefly discussed is the
idea of Pipeline swarming when the continuous delivery pipeline breaks. The swarming
idea closely relates to the Andon culture where we stop the line when something goes
wrong. The stop the production mentality where everyone swarms to fix any pipeline
problem until it’s fixed. When there is a problem with the continuous delivery pipeline,
the problem is swarmed and fixed, then the findings are integrated immediately into the
process or product as they are discovered. A key here is to remember that when the
217
pipeline breaks, we stop all work and swarm the pipeline, only after the pipeline is fixed
do we re-start our other production work. Without Open communication, it becomes
increasingly difficult for teams to swarm issues, fix incidents, and unblock the release
pipeline faster.
It’s all about Feedback Loops
The concept of DevOps feedback loops borrows heavily from a number of sources, some
of which go back sixty years. One of the first concepts of feedback loops was Plan, Do,
Check, Act "also known as PDCA", "The Deming Wheel" or "The Shewhert Cycle". "Plan-
Do-Check-Act" was first popularized in Japan in 1950 and 1951 during the lectures of a
Dr. W. Edwards Deming. The short version of these feedback loops is to plan out your
project, try it, examine the results and then incorporate any changes you need to make
based on the results.
A newer version of the feedback loop is the Lean Startup concept of build-measure-
learn feedback loops. The build-measure-learn process happens to go hand in hand with
Agile development since they both have the basic concept of figuring out a problem that
needs to be solved and then building a minimally viable product so that we can begin the
process of learning as quickly as possible. In Agile we do this through teams building out
features in an iterative manner, meaning that during each iteration they create a small
portion of the feature until they have built a minimally viable product. As the teams release
218
the feature in an iterative fashion, the entire process of releasing is in a sense the
measurement portion where starting with Test Driven Development all the way to the
monitoring phase the teams are measuring the feature and learning. A key portion of the
measuring and learning happens during an A/B testing phase where teams can receive
direct feedback from customers as too their satisfaction with the feature being tested. All
along the feedback loop, this method offers a scientific approach, introducing a systematic
way for teams to engage in product development.
Iteration 3
Company Dolphin
A. Company Epic creates fourteen User Stories and corresponding Specification by
Example scenarios to initially develop a new Membership Rewards Home Page and
places them in the backlog.
Story 5, 6, 7 and 8 in the third iteration.
C. The team gets together and flushes out all of the Specification by Example scenarios
for the four User Stories that were taken in during this iteration
D. The developers have a design meeting to flush out the details
E. The developers start building out the code in a TDD fashion using the Specification
F. The developers build the code using monitoring driven development so that the new
G. Another team member starts writing out the Functional automated tests in a TDD
fashion.
J. The Product Owner accepts User Story 5, 6, 7 and 8 during the Sprint Review.
“off”.
219
L. At the end of the third Iteration, the Login and Home Page functionality is pulled
from the artifact repository and loaded into a UAT environment and turned on so
that internal customers can try the new functionality.
N. In the feedback loop from UAT, the internal customers feedback is that they would
prefer that the rewards categories be displayed in a table format
O. As part of this iteration, the team also conducts an A/B test to determine what color
scheme is preferred by the customers.
P. All suggested changes from the internal customers and the results of the A/B test
are able to get through the Continuous Integration and Acceptance phases in order
to get them into the UAT environment.
Q. Move to Production Off section
1.
2.
3.
4.
220
Learning Point(s):
DevOps Orchestration Versus Automation
A common misconception in DevOps is that when we are discussing automating the

entire delivery pipeline and orchestrating all the automated pieces into a single process
we are discussing the same thing. There is actually a significant difference between
orchestration and automation, automation deals with single automated functions or
tasks, while orchestration involves arranging all of the individual automated tasks into a
single process. To understand the difference, we need to key on the concept that
automation is about codifying individual tasks and orchestration is about codifying entire
processes. Orchestration makes use of automation by using it as basic building blocks.
In DevOps, the concept of orchestration normally involves procedures or scripting tools,

which are merely a collection of commands used to deploy and connect an application.
The orchestration tools on the market take an organizations manual processes and derive
these processes into scripts. It is important to remember that though the orchestration
221
tool can fairly easily turn manual processes into scripts, the orchestration tool only
focuses on the process and doesn’t check the process outcome.
In DevOps, it’s necessary to use automation and orchestration together because they
can build standardized processes which allow for greatly shortened cycle time in order to
get code delivered to the end user. But as was mentioned at the beginning of this section,
it’s important to know the difference between automation and orchestration.
Quality Gates
One of the simplest ways to get started in improving quality and tackling technical debt
seems to also be one of the hardest to convince organizations and the developers inside
the organization. The idea of checking code quality upfront before checking code in would
seem like a logical way to develop software. By checking the code quality before it is
checked in, organizations would be able to prevent a large portion of the technical debt
they are incurring, but getting them to use a tool such as SonarQube and its Quality
Gates is much more difficult than most would imagine. SonarQube can be installed on
relatively modest machine to get started, and then be configured along with the developers
IDE so that the developer can at least do a code quality check before they check their code
222
into the source code repository. This is the simple way to get started with SonarQube,
after this it’s advisable to connect SonarQube’s Quality Gates into the Continuous
Integration process. So, what are these Quality Gates?
As SonarQube has matured they have taken the concept formerly known as alerts and
taken it out of Quality Profiles so that it could become Quality Gates. To imagine Quality
Gates, you need to think of them in two distinct ways. At one level, they are a collection
of the former alerts, and are instead now known as "conditions". At a high level, they are
a set of logical gates, stringing together all the conditions in the set of quality checks to
determine if a project can pass.
The new changes to Quality Gates now mean that the same criteria to all projects, across
languages. Now you can set it up once and apply it to each project specifically or set your
favorite gate as the default.
A/B Testing
223
Simply put, the ability to perform A/B testing in an automated fashion is a strategic
differentiator that DevOps organizations can possess over their competitors. A/B testing
is also one of the least known concepts among organizations looking at possibly
attempting or considering DevOps.
What is A/B Testing?
The basic concept of A/B testing is that it is a way to compare two versions of something
to figure out which performs better. A/B Testing is most often associated with websites
and mobile apps, the reality is that A/B Testing has been around for almost 100 years.
A/B Testing was introduced by a 1920s statistician and biologist named Ronald Fisher
who is credited with discovering the key principles behind A/B testing and randomized
controlled experiments. Others ran these experiments before Fischer, but he is the one
who figured out the basic principles and mathematics that made it a science.
How does A/B Testing Work?
To use A/B Testing you first have to determine what you wish to test. An example would
be whether users prefer clicking a button labeled Send or a link text labeled Send. Then
using one of the DevOps release strategies to put the experiment into production or maybe
an internal user’s environment where you would then evaluate the performance of each.
In this example, we want to see which option visitors to a web site click most often. You
run the test by selecting two sets of randomized users and then showing them based on
random selection either the button or link. From the analytical data, we determine which
option users preferred. It is important to remember that other factors could influence the
results, so this is why randomizing which users are in which group, you minimize the
chances that other factors will skew your results.
What Does A/B Testing Report?
In most organizations, some form of software will be used to automatically handle the
calculations and often there will be a statistician who handles interpreting the results.
A/B Tests will generally report out two conversion rates, the first for users who saw the
control version, and the other for users who saw the test version. Typical conversion rates
measure clicks and other user actions.
224
Mistakes Made in A/B Testing
The main error made when doing A/B Testing is being too eager and not letting the
tests run for a sufficient length. Many of the A/B Testing software products actually let
organizations watch the test results in real time, which leads to the urge to make decisions
too quickly. This over eagerness has been feed by the A/B software vendors who have
offered a type of A/B testing called real-time optimization, which allows organizations to
use algorithms to make adjustments as the results come in. This causes a problem in
that, due to the randomization, it highly possible that if you let the test run its full length,
you quite possibly could get a different result.
Managements Role in Agile and DevOps
225
Few DevOps improvement initiatives which organizations attempt to scale, often with
high expectations, will go on to fulfill their potential of having a major impact on
performance. The truth be told most of these attempts will fail, resulting in the companies
implementing them becoming disillusioned with the results. Yet, although there are many
examples of DevOps efforts that have failed, there are also examples of successful Agile
and DevOps implementations. So why do some these Agile and DevOps improvement
efforts disappoint? Some common reasons include an organizational culture that
discourages any change for example. But there are also some tangible management
causes of Agile and DevOps initiative failures.
Top-management support
The importance of top-management support goes far beyond the allocation of resources
to scale the Agile and DevOps initiative; it sets the priorities for the whole organization. If
the organization’s senior managers do not understand and show commitment to the
initiatives, it is only understandable that others will ask why they should do so. Usually
this is taken to mean that top management must:
1. Understand and believe in the benefits of the Agile and DevOps improvement
approach
2. Communicate the importance of these principles and techniques
3. Participate in the improvement process
4. Maintain a clear ‘strategic strategy’.
This last point is particularly important. Without thinking through the overall purpose
and long-term goals it is difficult for any organization to know where it is going. A strategy
is necessary to provide the vision, goals and guidelines which help to keep the Agile and
DevOps efforts in line with overall strategic aims. Specifically, the strategy should have
something to say about the competitive priorities of the organization, the roles and
responsibilities of all parts of the organization, the resources available, and the overall
philosophy (Continuous Improvement culture).
Senior managers may not fully understand the Agile and DevOps improvement approach
226
It is not difficult to find past examples of where senior management have used one or
more improvement approaches without fully understanding them. The details of Six
Sigma, lean, or ITIL for example, are not simply technology issues. They are fundamental
to how appropriate the approach could be in different contexts. Not every approach fits
every set of circumstances. So, understanding in detail what each approach means must
be the first step in deciding whether it is appropriate. Today many top managers are
enthralled with the idea of incorporating some Agile and DevOps practices in their
organization, but the reality is, Agile and DevOps are not a menu where you can pick and
choose what you like and don’t like.
Help the Agile and DevOps initiative keep their eye on the end goal
Agile and DevOps have, to some extent, become a fashion show with new ideas, tools,
processes and concepts continually being introduced as offering a cutting-edge way to
implement Agile and DevOps. There is nothing naturally wrong with this. It tends to
stimulate and refresh, through the introduction of these cutting-edge ideas. Without
them, things would stagnate. The problem lies not with new Agile and DevOps ideas, tools,
processes or concepts but rather with some teams becoming victims of these ideas, tools,
processes and concepts, where some new idea will entirely displace whatever went before.
Most new ideas have something to say, but jumping from one idea, tool to another will not
only generate a backlash against any new idea, but also destroy the ability to accumulate
the experience that comes from experimenting with each one (fail fast and learn from it).
Avoiding becoming a fashion show victim is not easy. It requires that those directing the
strategy process to take responsibility for a number of issues.
1. They must take responsibility for Agile and DevOps as an ongoing activity, rather
than becoming champions for only one specific Agile or DevOps idea, tool,
process or concept.
2. They must take responsibility for understanding the underlying ideas

behind Agile and DevOps. Agile and DevOps are not a recipe or painting by the
numbers. Unless one understands why Agile and DevOps improvement ideas are
supposed to work, it is difficult to understand how they can be made to work
properly.
3. They must take responsibility for understanding the ancestor to a new

Agile and DevOps idea, tool, process or concept, because it helps to understand
227
it better and to judge how appropriate it may be.
4. They must be prepared to adapt new ideas so that they make sense
within the context of their own Agile and DevOps initiative. One size rarely fits all.
5. They must take responsibility for the (often significant) education and
learning effort that will be needed if Agile and DevOps are to be successfully
implemented and exploited.
6. They should avoid the over-exaggeration and hype that many new ideas
attract. Although it is sometimes tempting to exploit the motivational
pull of new ideas through evangelism, carefully thought-out plans will
always be superior in the long run, and will help avoid the inevitable
backlash that follows over-hyping a single approach.
Iteration 4
Company Dolphin
Story 9, 10 and 11 in the fourth iteration.
B. The team gets together and flushes out all of the Specification by Example scenarios
for the three User Stories that were taken in during this iteration
D. The developers start building out the code in a TDD fashion using the Specification
fashion.
228
G. The Specification by Example scenarios for this iteration call for Security,
Performance load and Front-End Performance testing. The team starts creating
these tests from the Specification by Example scenarios.
J. The Product Owner accepts User Story 9, 10, and 11 during the Sprint Review.
“off”.
L. At the end of the fourth Iteration, the Login and Home Page functionality is pulled
N. Move to Production Off section
1.
2.
3.
4.
229
Learning Point(s):
Striving for Green
One of the cultural changes that occurs when the light build goes on in organizations
attempting to implement DevOps is when teams start what is termed either “striving for
green” or “push on green”. In short, this concept is means that deployments are
automatically executed when all tests pass. This is synonymous with "Continuous
Deployment." The accepted method for automating deployments to environments is to
automate testing.
If your automated testing is rigorous enough, you can deploy any code for which the
tests pass. In very mature organizations this can lead to Continuous Deployment all the
230
way to the production environment. In reality, striving for green or push on green has a
deeper meaning. When teams start striving for green it means they have fully bought into
the concept of automation, meaning automating everything from environments, to builds,
to testing, to deployments. Striving for green culturally means that teams are now fully
committed to seeking out and finding all areas that can be automated in order to minimize
as much of the manual toil as possible. A strong strive for green culture will automate the
vast majority of the overall development and delivery process so the team can focus on
specific, value-added areas. In order to achieve this, an investment needs to be made in
achieving the working practice of automating the norm.
DevOps = Continuous Learning Organization
231
Understanding Continuous Improvement means that organizations need to learn that
the ability to improve, especially on a continuous basis, is not something which always
comes naturally inside many organizations. Organizations need to be aware that there are
specific abilities, behaviors and actions which need to be consciously developed if
improvement is to be sustained over the long term. Below we will highlight the
organizational abilities that allow for the adoption of continuous improvement.
Organizational Ability Behaviors

Getting the CI habit • People use formal problem-
Developing the ability finding and solving cycle such
to generate sustained as feedback loops
involvement in CI • People use simple tools and
techniques like double loop
Learning
• People use simple
measurements to shape the
improvement process
• Individuals and/or groups
initiate and carry through CI
activities – they participate in
the process
• Ideas are responded to in a
timely fashion – either
implemented or otherwise dealt
with
• Management supports the CI
process through allocation of
resources
• Management recognizes in a
formal way the contribution of
employees to CI
• Management lead by example,
becoming actively involved in
design and implementation of
CI to help drive out fear
• Management supports
232
experiments by not
punishing mistakes, but
instead encouraging
learning from them – fail
fast and learn from it
Spreading the word • Implementing cross

Generating the ability to functional teams
move CI activity across • People understand and
organizational boundaries via share a holistic view of
DevOps DevOps (process
understanding and
ownership)
• People are oriented towards
internal and external
customers in their CI
activity – Never break the
customer – internal or
external
• Relevant CI activities
involve representatives
from different
organizational levels
CI on the CI system • The CI system is
Generating the ability to continually monitored and
manage strategically the developed
development of CI • There is a cyclical planning
process whereby the CI
system is regularly
reviewed and amended
• There is periodic review of the CI
system in relation to the
organization as a whole
• Senior management makes
available sufficient
233
resources (time, money,
personnel) to support the
continuing development of the
CI system
• The CI system itself is
designed to fit within the
current structure and
infrastructure
• When a major organizational
change is planned, its potential
impact on the CI system is
assessed
Walking the talk • The ‘management style’ reflects
Generating the ability commitment to CI values –
to articulate and Deming’s 14 Points
demonstrate CI’s values • When something goes wrong,
people at all levels look for
reasons why, rather than blame
Individuals – pipeline swarming
• People at all levels demonstrate
a shared belief in the value of
small steps and that everyone
can contribute, by themselves
being actively involved in
making and recognizing
incremental improvements
Building the learning • Everyone learns from their
organization experiences, both good and bad
Generating the ability to • Individuals seeks out
learn through CI activity opportunities for
learning/personal development
– Dojos, Centers of Excellence
• Individuals and groups at
all levels share their
learning – single source of
information (GIT)
234
• The organization captures and
shares the learning of
individuals and groups
• Management accepts and acts
on all the learning that takes
place
• Organizational mechanisms are
used to deploy what has been
learned across the organization
Wagner, John A, and John R. Hollenbeck. Management of Organizational Behavior.

Prentice-Hall, 1995.
Improvement is Learning
What is critically important to realize is that improving is a learning process, and it is

crucial that the improvement is organized so that it encourages, facilitates and exploits
the learning that occurs during improvement.
235
Created a New DevOps Department? Go Back!
If you do a simple search on LinkedIn for DevOps, what you discover is that it seems
everyone today has an official title such as DevOps Engineer. When you look around a
little more, you also discover that a large portion of organizations attempting DevOps have
decided the path to follow is to setup a brand new permanent DevOps department. Is this
the correct path to follow for DevOps? If you listen to what organizations who can be said
to be truly practicing DevOps have to say or the top names in the DevOps movement, the
answer is a resounding no!
Let’s explore some of the problems that may come out of creating a brand new
permanent DevOps department with official titles, some may have been mentioned by
others and some you may not have considered.
First, let’s take a look at the foundations that DevOps is based on, you have probably
heard DevOps is based on Lean/Lean Manufacturing and a few individuals may have
heard Dr. W. Edwards Deming’s work mentioned. The reality is that you should be paying
very close attention to Dr. Deming’s work, specifically Point 9 of his 14 Points work. In
236
Point 9, Dr. Deming calls for companies to “break down barriers between departments”.
Dr. Deming goes on to explain that everyone from people in research, design, sales, and
production must work as a team, to foresee problems of production and in use that may
be encountered with the product or service. Does any of this sound familiar with DevOps?
Aren’t we trying to foresee problems in production by automating the processes to control
variation, creating feedback loops etc. etc….? Isn’t DevOps using feedback loops,
automated testing and deployment strategies to uncover issues that customers may
encounter with the product or service.
In looking at today’s environment of rapid change, we need to understand that in the

past we saw stable, slowly changing environments, where learning and adaption could
proceed slowly. In these stable, slow changing environments, trial and error was an
effective and possibly the most appropriate, learning strategy. In the unstable, rapidly
changing environments of today, such as those caused by rapid technology change,
explicit, systematic learning must be the linchpin of strategies to keep organizations
adapted to their environment. If you are not careful how you organize yourself, all
innovations end up being incremental. Therefore, it is even more crucial that you organize
yourself for a more disruptive type of innovation—and you can’t always expect it to come
out of an individual team/department. As long as you continue to have different
departments (such as a DevOps department), you will continue to have silos—and that
hinders your agility, innovation potential and organizational learning.
While information technology can enhance individual learning within the organization
in many ways, a more important task is to capture that knowledge to the benefit of the
organization. This occurs in several ways. One way is by improving coordination,
communication, and collaboration among individuals so that the expert knowledge and
learning of individuals within the organization can be transferred from the individuals
with knowledge to individuals who need or can take advantage of that knowledge.
What this sets up is the argument made by most of the DevOps leaders and
organizations that have successfully adopted DevOps, which is that it’s critical to not
follow the path of creating a new DevOps silo with formal titles, and instead make use of
methods such as Dojo’s to disseminate the knowledge of individuals to others inside the
organization and not trap it inside a new silo.
A second way is by developing methods to capture and preserve individual experts

learning outside of memories and experiences of those experts. While many organizations
recognize that their most important assets walk out the door at night, few are
237
systematically seeking out ways to capture and preserve that expertise. This is where the
use of automation is so critically important in DevOps, organizations must take the
knowledge that is the minds of individuals in terms of how to setup infrastructure,
architecture, processes, tooling and transfer that into code that can be used to automate
the delivery process. By transferring this knowledge into code, organizations can extend
this individual learning to the broader organization, instead of it being contained inside a
silo. Having a DevOps department do the automation and give others a button to push is
in no way transferring knowledge.
Lastly, another issue with creating a permanent DevOps department is that it stifles
the DevOps attempt to bring back an Engineering Culture or Startup Mentality. When you
silo up a group of individuals doing all of the what’s termed cool work on new and exciting
technologies, what you don’t hear from other parts of engineering is people talking
energetically about technology and excitedly discussing the possibilities for these
technologies. People want to be proud of their engineering skills and achievements, not
feeling like they are locked into a specific sector with no opportunity to contribute.
So, what have the organizations who didn’t create DevOps departments realized that
organizations following the permanent DevOps department creation path haven’t? They
have realized that DevOps is about attempting to institutionalize their processes, so they
can capture learning of individuals/teams and make learning more broadly available to
others in the organization who can also derive benefit from this learning. Remember,
Organizational Learning is our ultimate goal and Agile and DevOps are simply vehicles to
help us get there, just like tools and methodologies are vehicles to help us with Agile and
DevOps.
238
Iteration 5
Company Dolphin
Story 12, 13, 14 and 15 in the fifth iteration.
for the four User Stories that were taken in during this iteration
fashion.
I. The Product Owner accepts User Story 12, 13, 14, and 15 during the Sprint Review.
J. Once the code successfully navigates the Acceptance phase and the Product Owner
“off”.
K. At the end of the fifth Iteration, the Login and Home Page functionality is pulled
L. The team now watches the telemetry that is providing a feedback loop from the UAT
M. At the end of this iteration the company decides to test their Andon Cord.
N. Move to Production Off section
239
1.
2.
3.
4.
Learning Point(s):
Andon Culture
Andon Cord is a Lean manufacturing principle and tool used to notify management,
maintenance, and other workers of a quality or process problem. The concept revolves
240
around a device incorporating signal lights to indicate which assembly line workstation
has a problem. Normally alerts are activated manually by a worker using a pull cord
(Andon cord) or button or may be activated automatically by the production equipment
itself. The system may include a means to stop production so the issue can be corrected.
The Andon System was developed as one of the principal elements of the Jikoda quality
method pioneered by Toyota as part of the Toyota Production System (TPS) and has
become part of the lean manufacturing approach. Andon cord gives the worker the ability,
and moreover the empowerment, to stop production when a defect is found, and
immediately call for assistance.
There are stories about individuals from companies who have toured Toyota's facilities
as part of the Kaizen Institute tours who are shocked when they are introduced to Toyota's
Andon Cord concept. They are usually in shock that someone on the factory floor can pull
a cord and possibly shut down the entire factory. The question is always how much does
this cost and why would you allow an individual to have this kind of power. The answer
from Toyota is always the same, it costs about $1 million dollars to shut down the entire
assembly line, but if they allowed the line to continue it could cost them millions of dollars
down the line.
What is really important in understanding Andon Cord from Toyota is the underlying
reason they created it, In Japanese culture, the importance of harmony is critically
important. Because of the overwhelming need for harmony, people often won’t naturally
speak up. They might be more willing to cover up a problem than to really fix it. So, the
Andon Cord is a mechanism that makes it easier for people to speak up. Because it
“doesn’t come naturally,” Toyota needed a system to make it possible.
So how does this apply directly to DevOps and software development? The reality is that
if we look closely at most software development projects, we have all probably seen the
following. During the development of the project individuals openly talk about all of the
bugs in the application, that it isn’t what the customer really wants and its being rushed
and the technical debt is piling up. Even though everyone involved is openly talking about
the issues, the project just keeps moving forward without dealing with any of these issues.
This is where DevOps comes in and can incorporate the Andon Cord concept. There are
two specific areas that come to mind, first if an organization has truly flipped the testing
pyramid and put full automated testing in place in conjunction with Continuous
Integration, Specification by Example and a tool such as SonarQube, this can be the first
241
place the Andon Cord concept can be employed. By forcing all new code to run the gauntlet
of full automated testing (driven by Specification by Example) and SonarQube’s quality
gates you are making sure it meets the expected behaviors specified by the customer and
the code quality is in line with expected standards.
The second area DevOps can function as a sort of Andon Cord is with A/B Testing.
When an organization has put into place a fully automated delivery pipeline they are able
to quickly get code out to subsets of their customers (often the same day the code was
developed) in order to create Feedback Loops that enforce that they are building what
their clients wants. There are other ways that Andon Cord is used with DevOps and
companies such as Amazon and Netflix have truly incorporated Andon Cord into their
culture.
DevOps Metrics
242
There are a number of metrics that DevOps borrows from Lean, such as cycle time, lead
time, WIP, MTTR and a few others. In order to determine how effective an organization is
on their DevOps journey; the following metrics should be used to measure an
organizations DevOps effectiveness.
How often do you Deploy?
This metric is exactly what it says, how often do you deploy a new version of an
application or service. The frequent deployment of applications or services suggests that
the organization may have an effective continuous improvement practice in place.
Frequent deployments also are a key in gaining business buy in.
Deployment time
This measures the amount of time that it takes to promote an application or service
from one environment to the next. A proper deployment in DevOps assumes an application
or service that is loosely coupled to be able to deploy without impacting the entire system.
Quality verification time
This is the amount of time that it takes for test automation to run and verify a new
application or service. This time should continue on a downward slope if teams are focused
on striving for green, meaning they are looking to continuously improve through
automation.
Continuous Integration verification frequency
This is a key indicator of the organizations automated testing coverage progress.

Without a high Continuous Integration verification frequency, the organization is most
likely still handling builds and the corresponding testing in a manual manner. Having a
manual build or testing verification, means the release pipeline process will never have a
high level of confidence in the code moving through it.
Pipeline success rate
This metric tracks the success rate of iterative code releases through the delivery
pipeline, this metric tells an organization if they have built high quality in the various
243
steps in the release pipeline. This metric indicates if the organization has successfully
dealt with “Drift” in the delivery pipeline and all of the components of automated testing,
such as data management, test coverage and have they shifted left with testing (are they
performing security, performance, consistency in the early stages of the pipeline).
Pipeline up time
This metric is an indicator of how often the delivery pipeline is up, which is an indicator
of the organizations efforts towards automated testing, code quality, environment
provisioning and Drift.
Specification by Example coverage ratio
This metric tracks the percentage of traceability between requirements and tests, which
leads to higher confidence levels organization wide for code moving through the delivery
pipeline.
Application/Service Feature usage
This metric is a key indicator of the DevOps feedback loops that the organization has
put into place. Low usage signals a breakdown in communications in one of the channels
between, customers, the business and IT.
Mean time to restore service (MTTRS)
This metric measures the average time it takes to restore a service or a function when
the disruption is due to a defect that requires a fix to be developed. A high MTTR could
indicate issues in the feedback loops, effectiveness of the delivery pipeline and the
organizations ability to react to change.
Quality Gate Detection
This metric is a key indicator of the code quality being released through the delivery
pipeline, which in turn results in lower technical debt.
244
Artifact Repository
A key part of DevOps is building a delivery pipeline, which is capable of deploying

development versions of your binary artifacts to a component repository (e.g., Nexus) and
to a target environment (e.g., deploying a web application to a web container). An example
would be a Maven-based build system where you want to share new development versions
(in Maven these are so called SNAPSHOT versions) of your artifacts.
Once we verify the code satisfies the requirements, you will then want to freeze this
specific version in version control (often known as tagging or labeling). In other words, a
baseline is created that expresses a release candidate that is a specific version of the
software for later reuse. Later, a release build is created. A release build promotes an
existing release candidate that will be eligible for production. It prepares release artifacts
(in Maven, for example, by removing the SNAPSHOT in version elements) and deploys
these artifacts to the component repository. Nonfunctional requirements (NFR) are
explicitly tested again before the release is staged to production. Of course, this process
is only a small subset of an overall pipeline.
245
Iteration 6
Company Dolphin
Story 16, 17 and 18 in this iteration.
for the three User Stories that were taken in during this iteration
C. The developers have a design meeting to flush out the details.
fashion.
I. The Product Owner accepts User Story 16, 17, and 18 during the Sprint Review.
J. Once the code successfully navigates the Acceptance phase and the Product Owner
“off”.
K. At the end of the fourth Iteration, the Login and Home Page functionality is pulled
L. The team now watches the telemetry that is providing a feedback loop from the UAT
M. Move to Production Off section.
246
1.
2.
3.
4.
Learning Point(s):
Dojos and Centers of Excellence
“The Target Dojo.” Target Enterprise Coaching, dojo.target.com/.
The idea behind Dojos was first introduced by Target. The idea for Centers of Excellence
has been around for many years in various industries. The basic concept for both Dojo’s
and Centers of Excellence are similar. Both Dojo’s and Centers of Excellence normally
have a dedicated space in which subject matter experts in various areas reside.
Individuals and teams looking to learn from these experts can move to the space where
the subject matter experts are located in order to have access to these resources. The time
spent located with the subject matter experts can range from days to weeks to a month or
more. Inside the space, subject matter experts can move from individual to individual and
team to team in order to support their training and learning needs.
247
As discussed in the Organizational Behavior section, by having individuals and teams
spend time in Dojo’s and Centers of Excellence they see better results due to the fact that
they are in a social environment that facilitates learning.
Security and Compliance
248
Traditionally, the development side of an organization has viewed security and
compliance as a very rigid group that has interfered with the DevOps effort inside the
organization because of their calls for separation of duties and documentation
requirements. On the other hand, the security personnel/teams have viewed the
development side as very risky, because of the increased number of software releases,
which have been seen as a risk to governance and security.
What organizations who have moved forward with DevOps have discovered is that the
fears of the security personnel have been mistaken, instead it has been shown that
DevOps practices actually mitigate potential security problems, discover issues faster, and
address threats more quickly. These findings have led to more and more security teams
adopting automation and DevOps practices. Many individuals/teams in the infosec space
are increasingly coming to view DevOps as their security blanket in order to better enable
and enforce security, compliance, and audit requirements, which has flipped the
traditional view of DevOps from being a threat to a resource.
In DevOps currently, organizations are more and more aligning their InfoSec with their
DevOps initiatives and seeing security requirements becoming an integral piece of their
DevOps practice.
DevOps Security and Compliance Benefits:
Security testing shifted left - DevOps organizations have incorporated

automated security testing early in the delivery pipeline and not as an afterthought just
prior to delivery to production or worse yet after releasing to production.
Drift Elimination - The more tests and processes that are automated means the less
chance organizations have of introducing security flaws due to human error.
Pipeline lockdown - The delivery pipeline becomes a locked down process that the
organization controls in all phases of delivery.
Coordination - Through the incorporation of security tools and tests as part of the delivery
pipeline, InfoSec becomes a component of the delivery pipeline.
MTTR Improvement - When the unfortunate security issue is detected, DevOps accelerates
the lead time, so that organizations can develop, test, and deploy a patch/update more
quickly.
249
Enable developers without risking governance - The automated delivery of environments
and data allows developers a self-service opportunity while at the same allowing for access
control and compliance.
Infrastructure code security - DevOps requires that infrastructure code be stored in an

SCM just like application code, which enables the creation of manageable systems which
are consistent, traceable, and repeatable.
Superior compliance reporting- A DevOps automated delivery pipeline has access to a

tremendous volume of information that is automatically logged in great detail. As this
information is logged, it is in reality becoming the audit trail, security log, and compliance
report, all of which are produced automatically, with no manual steps or long hours
tracing the processes and actions to produce the compliance report.
Evolutionary Development/Architecture
Sometimes in the hast to develop new ideas and concepts we tend to disregard prior
research that has already proved what we concluded is a good idea. This is the case for
Evolutionary Development/Architecture which many in
250
the DevOps community are touting as the newest and latest trend for development. If we
simply did a little research, we would discover that Harvard Business School professor
Alan MacCormack and his colleagues Marco Iansiti and Roberto Verganti through their
two year research study uncovered four practices that lead to successful software
development in the year 2001.
In their two-year research study, they analyzed data from 29 completed projects and
identified the characteristics most associated with the best outcomes. (You can read in
detail their results in “Four Software-Development Practices That Spell Success.”) What
they discovered was that successful development was evolutionary in nature. Companies
who would first release a low-functionality version of a product to selected customers at
a very early stage of development and then layer on top of it in an iterative fashion, with
the design allowed to evolve in response to the customers’ feedback. This approach
contrasted with traditional models of software development at the time and their more
sequential processes flows. They found that this evolutionary model has been around for
several years, but this was the first time the connection has been demonstrated between
the practices that support the model and the quality of the resulting product.
251
Microservices
The DevOps movement has brought on the advent of microservices, which is an

architectural pattern that composes distributed applications from separate deployable
services that are designed to perform a specific business function and communicate via
web interfaces. In DevOps, we see a pattern where teams encapsulate individual pieces of
functionality inside microservices and thus are able to build larger systems by combining
the microservices much like you would with building blocks. It’s important to understand
that microservices employ what is termed the open/closed principle, which means they
are open for extension, and closed for modification.
The use of microservices in DevOps provides a number of advantages to organizations

over traditional large-scale intertwined architectures. One principle advantage of
microservices is that they remove single points of failure because an individual
microservice should not impact any other part of an application. Microservices also allow
organizations to scale out the functionality the microservice provides independently from
other functionality in order to increase capacity. The beauty of microservices is that
organizations can simply extend the functionality of an application by simply creating a
new microservice and not affect other parts of the application. This ability to scale
independently also has tremendous implication on Technical Debt.
Another advantage to the use of microservices is that it can increase productivity when
used in conjunction with DevOps. Microservices are built, versioned and deployed just
252
like traditional applications using Continuous Integration and Continuous Delivery. The
move to cloud-based applications has also enhanced the use of microservices, since
organizations can make use of patterns such as event-driven programming.
DevOps is also seeing an increasingly used patterns where containers are clustered to
implement the organizations microservices.
Containerization
Defining Containerization
The strategy of containerization is that we are using an operating system-level

virtualization for running distributed applications without having the need for launching
a virtual machine for each application. This involves running a multitude of isolated
systems on a single host and accessing a single kernel. All of the environment variables,
files, and libraries necessary to run the desired software are part of the key components
of application containers. Containerized software or containers are available for
both Linux and Windows-based apps. Containers generally allow you to isolate software
from their surroundings, making it a segregated piece of software. Containers also have
253
the unique ability to bring out differences between development and staging environments
(identify Drift) thereby helping eradicate conflicts between teams running different
software on the same platform.
Container Advantages
Containerization allows organizations to gain efficiencies in memory, CPU, and storage.

These are benefits that Containers compared to the traditional virtualization approach.
Another advantage is that application containers don’t have overheads that are required
by VMs, and because of this reason, it is attainable to help numerous more holders on
the same infrastructure (Allows for us to automate our Delivery Pipeline). The use of
Containers allows us to package application code, configurations, and dependencies into
easy to use building blocks that deliver developer productivity, environmental consistency,
operational efficiency, and version control (Again eliminating Drift). One of the industry
leaders, Docker containers, allows us to run on a single machine that shares the
machine’s operating system kernel. Since containers share the same operating system
kernel as the host, containers can be more efficient than VMs that require separate
operating systems. The host operating system also constrains the container’s access to
physical resources like CPU and memory.
254
Cloud
Infrastructure as a service (IaaS) and cloud platforms are a critically important part of
the DevOps ecosystem. The use of cloud platforms allows companies to buy memory, data
storage, server time and computing power as they need it. This lets companies and teams
focus on building their product/service instead of building infrastructure from scratch.
The Cloud is a direct response to the need for agility. At the start, organizations saw
the cloud primarily as a way to save money. These same organizations have since come to
realize that its real value lies in reducing waste that impedes speed. Very few companies
would identify data center operations as part of their core value proposition. The
introduction of cloud services has allowed IT departments to shift their focus away from
commodity work such as provisioning hardware or patching operating systems and spend
it instead on adding business-specific value.
255
Iteration 7
Company Dolphin
A. The business makes the decision to turn on the new Rewards Program functionality
in production.
B. The business makes all the necessary marketing announcements.
C. The business receives feedback from the internal customers that they would like to
login into the Rewards Program with their LinkedIn login. The business creates a
new story for the LinkedIn login and moves it to the top of the backlog.
D. The team starts their next sprint by taking in new stories from the backlog including
the LinkedIn login story, while also continuing to monitor the Rewards Program
functionality in production.
E. The team gets together and flushes out all of the Specification by Example scenarios
for the three User Stories that were taken in during this iteration.
F. The developers have a design meeting to flush out the details.
G. The developers start building out the code in a TDD fashion using the Specification
H. The developers build the code using monitoring driven development so that the new
I. Another team member starts writing out the Functional automated tests in a TDD
fashion.
J. After the third day, developers start submitting Pull requests for code reviews.
K. The Continuous Integration automatically starts picking up the Pull requests after
L. The Product Owner accepts the User Story’s during the Sprint Review.
M. Once the code successfully navigates the Acceptance phase and the Product Owner
gives the green light, it is moved into Production.
256
1.
2.
3.
4.
Learning Point(s):
DevOps Release Strategies
257
The DevOps movement through the emphasis on incremental releases which were
discussed in an earlier section has allowed for the incorporation of new release strategies
that serve to allow for rapid deployment to production, increased quality, and strategic
advantages through the use of A/B testing and live statistical sampling.
One of the tremendous advantages to using one of the DevOps release strategies along
with incremental releases is that is allows for a decoupled deployment and release. What
this means is that we no longer are tied to the traditional model of having major new
releases to customers every year or couple years as the software industry has followed in
the past. Now organizations can break out individual features or group features and send
them to production at any time without waiting for other features to be packaged up into
a major release.
There are some major strategic advantages to the decoupled deployment and release
model. Organizations can make use of Pilot Releases which give them the ability to market
test the new features with real customers at a low cost. The advantages of this are rather
simple, why not market test new features or ideas with real customers in production like
environments. The idea behind this is that you roll out the new feature in the pilot so that
it exists in parallel to production. The pilot can be made available to selected customers
who can use the feature in their actual business operations. The pilot customers can be
made up of internal users, external customers, or a statistical sample subset of customers
using A/B testing.
By getting this pilot feedback, the development team can continue to improve and
extend the feature. When it is determined that the customers are satisfied with the feature,
it can then be deployed to production for the entire customer base to use. In the case
where the feature does not meet the customer’s needs, it can simply be switched off and
not delivered to the entire customer base.
Because DevOps brings about much shorter feedback cycles between customers and
developers, the result is a more flexible and reactive systems evolution. Because of the
ability to use pilots, organizations can now ask customers or internal users to become
true collaborators. The collaborators can become involved in the design of the feature
during the feature creation process.
DevOps calls for telemetry to be embedded in the software the organizations develop in
order to collect data on feature usage. Software, such as web applications, mobile apps,
258
and internal applications, enable real-time access to usage data and even offer real-time
analytics. This allows the development teams to act rapidly on this data and perform
analysis and make adjustments. It’s also important for organizations to monitor feature
usage.
Some General DevOps Release Strategies:
Feature Toggles
The idea behind feature toggles is to be able to deliver the complete code to production
but use switches to determine which feature will be made available in production. In order
to enable the switches, configuration files are most often used. Through the use of feature
toggles, teams are able to develop on the mainline (Branches are not needed) and then
ship the completed code to production.
Dark Launching
Dark launching involves the deployment of a feature to a statistical sample of customers

in production before releasing the functionality to all users. Dark launching in essence
decouples the deployment of a new version of software from the release of the features
within it. Organizations using Dark Launching can deploy new versions of their software
continuously, regardless of which features are available to which users. By releasing to a
statistical sample of users in production, it provides the ability to find any bugs more
easily, before releasing to all users.
Blue-Green Deployment
Another DevOps release strategy is blue-green deployment. At the heart of this strategy
is the deployment of a new version of the application side by side with the current
production version. To switch over to the new version or roll back to the current
production version, a simple change to a load balancer or router setting is changed. The
idea behind Blue-green deployment is to ensure that the organization has two production
environments that are as similar as possible. At any one time, one of them (the green
environment) is live. What this allows is while the organization is preparing a new release
of software to production, they can conduct their final testing in the blue environment.
When the organization determines that the software is working in the blue environment
as expected and tested, configurations are then completed, the router is then redirected
to send all incoming requests to the blue environment. After this the green environment
259
is now no longer production and can be used as the blue environment to prepare the next
release.
Canary Deployment
Canary releasing is a slight variation of the blue-green deployment strategy that is often
used when running a cluster of servers. The idea here is that instead of upgrading a whole
cluster to the latest version all at once, you instead deploy incrementally. The new release
is directed at a chosen subset of production systems, of which only known users, or a
specific user group are routed to. If the deployment is successful, the release can be
deployed to more and more users, and finally to all available users.
Simian Army
The Simian Army is an open-source IT tooling developed by Netflix to improve

availability and reliability by handling various cloud computing issues and challenges. It
is built with a variety of Simian Army Monkey tools to help ensure network health,
promote efficient traffic and locate security problems.
The Simian Army runs on the Amazon Web Services (AWS) infrastructure. In creating
the Simian Army, Netflix began with Chaos Monkey, described as a tool that targets
individual production instances and disables them to provide more stable and consistent
260
operations. Netflix then introduced Latency Monkey, which simulates partial network
downtime, and Conformity Monkey, which enforces network consistency. Janitor Monkey
eliminates excess digital clutter in a cloud environment, and Doctor Monkey performs
network health checks.
Other tools, like Security Monkey, 10-18 Monkey and Chaos Gorilla are also part of this
ground-breaking IT system.
Latency Monkey induces artificial delays in our RESTful client-server communication

layer to simulate service degradation and measures if upstream services respond
appropriately.
Conformity Monkey finds instances that don’t adhere to best-practices and shuts them
down.
Doctor Monkey taps into health checks that run on each instance as well as monitors
other external signs of health to detect unhealthy instances.
Janitor Monkey ensures that our cloud environment is running free of clutter and
waste.
Security Monkey is an extension of Conformity Monkey. It finds security violations or

vulnerabilities, such as improperly configured AWS security groups, and terminates the
offending instances.
10–18 Monkey detects configuration and run time problems in instances serving
customers in multiple geographic regions, using different languages and character sets.
Chaos Gorilla is similar to Chaos Monkey but simulates an outage of an entire Amazon
availability zone.
261
False Prophets
An issue that must be brought up in the book because it is having such an adverse
effect on the DevOps movement is the case of false prophets. In terms of DevOps, false
prophets come in three forms, first there are those blogging, running meetups, speaking
and writing about DevOps, but who have absolutely no idea about any of the
fundamentals DevOps is built on. You can usually spot these false prophets because their
entire message about DevOps will be almost entirely based on tools with little to no
message about culture or the foundations DevOps is built on.
The second set of false prophets are recruiting firms/outsourcing firms. Both of these
groups will claim that they can solve all of your DevOps problems if you will just allow
them to handle the resource issues for you. Recruiting firms will promise that they can
find you as many DevOps engineers as you need (as discussed, there should not be any
DevOps Engineer titles). The problem? There simply are not individuals in the market
262
place who have all of these skills organizations are looking for. This reminds me of working
with various organizations on automated testing. I would often be asked to sit in on
interviews of potential automated testing candidates who the recruiting firm had promised
had impeccable qualifications. As was almost always the case, within a couple of minutes
of the interview you realized the candidate didn’t have any coding skills and had only
performed record and play automation. Remember, almost none of the recruiters even
have the slightest idea what DevOps even is, so how are they going to find you qualified
candidates. When we look at outsourcing firms you need to ask yourself this simple
question, if DevOps is about changing your culture in order to be successful, then how
are you going to outsource to a firm that most likely hasn’t undergone the cultural changes
necessary to do DevOps themselves? As organizations discovered during the Lean
Manufacturing revolution, they had to provide extensive training of their employees and
did this from within, they didn’t look for someone to solve their resource skills for them.
The last set of false prophets are the tool vendors. All you have to do is go to almost any
DevOps conference, look at an online ad or listen to a sales rep and you will discover that
they are promising that if you will just install their tooling all your DevOps be solved. As
we have stated over and over in the book, DevOps is about cultural change and no matter
what tools you buy, they can’t solve your cultural problems.
Let’s Summarize Orca’s and Dolphin’s Probable Outcomes:

1.
2.
3.
4.
263
Conclusion
As we have shown in the book, the journey to Agile and DevOps can be extremely
difficult if the Agile and DevOps leaders and the organization are not willing to confront
the difficult structural and cultural challenges they will face on their journey. When we
look at our two companies Orca and Dolphin, we can see many of the learning points
discussed in the book and get a sense of their chances of survival. While nothing is
assured in the journey to Agile and DevOps, the leaders and employees at Dolphin have
at least taken a number of concrete steps in the right direction and have a much better
chance of success then what we see with their counterparts at Orca.
264
Automate all repeatable processes, 200
1 automated accessibility testing, 100, 243
14 Points, 7, 17, 22, 28, 34, 68, 105, 265, automated testing, 18, 20, 66, 72, 83, 91,
267 97, 99, 100, 118, 188, 204, 222, 225,
241, 242, 243, 270, 285
A automating work stations, 198
A/B testing, 20, 108, 181, 245, 252, 253,
B
254, 281, 282
A/B Testing, 252, 271 backlog grooming, 114
Acceptance Driven Development, 117 backlogs, 114, 214
Acceptance Test-Driven Development, Bargaining and Negotiation, 155
117 Barriers to Effective Communication, 9,
Adaptive Change, 151 144
adjustment ‘costs’, 210 Baseline, 129
adjustment issues, 210 basic business methods, 64
agent for cultural change, 118 basic human survival instinct”., 162
Agile, 7, 8, 10, 15, 19, 23, 24, 25, 30, 33, Be Committed to the Cause, 218
42, 49, 50, 51, 55, 56, 57, 59, 60, 61, Behavior Driven Development, 117
70, 71, 72, 73, 75, 76, 80, 81, 84, 96, behavioral level, 63, 64
99, 100, 101, 103, 108, 112, 113, 114, Bernoulli's Principle, 7, 41
118, 119, 129, 142, 146, 151, 162, 163, Better operational control, 215
164, 165, 173, 178, 179, 182, 183, 185, blame game, 46, 136, 137, 202
186, 188, 189, 202, 203, 208, 214, 221, Blame Game, 9, 102, 135, 148, 153, 155,
222, 224, 225, 227, 234, 243, 245 214
Agile artifacts, 108 blameless, 137
Agile Best Practices, 112 Blameless Post Mortems, 23, 24
Agile founders, 71, 185 Blue-Green Deployment, 283
Agile methodology, 30, 42, 49, 50, 55, 70, bottlenecks, 36, 123, 214
71, 96, 112, 113, 178, 179, 202, 224, Breaking Down the Silos, 226
234 Bruce Chew, 210
Agile Upstream and Downstream, 70 build in quality, 39
Amazon, 17, 21, 271 Build Relationships, 218
Amazon’s, 17 BuildID, 130
Andon, 12, 21, 39, 244, 268, 269 Building the learning, 266
Andon Cord, 21, 268, 269 Bureaucratic Inertia, 154
Andons, 39 Business analyst, 117
Anticipate the Emergence of Negative Business Analysts, 24
Politics, 148 business core, 111
API’s, 242
C
Application/Service Feature usage, 274
Appreciation of the System, 33 Canary Deployment, 284
centers of excellence, 22 conversion rates, 253
Ceremonies, 157 Conway’s Law, 9, 125, 126
change agent, 149, 150, 176, 217 Cosmopolitans, 146
Change Management, 9, 75, 132, 133 critical behaviors, 118
Changing the Culture, 7, 63 cross browser/os testing, 100
check and balance, 137 cross cultural, 64, 65
Chef, 199 Cross Cultural Communication, 10, 157
cherry picking, 71 cultural barriers, 188
Chevrolet Cruze, 200 Customer Focused, 235
Chevrolet Volt, 200 Cycle time, 181
CI on the CI system, 265 Cycle Time, 39
CI process, 97, 263
Code Reviews, 237 D
common verbiage, 117 Daily Standups, 71
Communicate Openly, 147 Dark Launching, 283
Competency, 218 Data files, 131
compliance reporting, 279 Data models, 129
Comprehensive Change, 150 Dealing with Fear, 9, 147
conditions, 252 Decision Making Illusions, 9, 141
Configuration files, 130 decoupled deployment, 282
Confront Political Game Players, 148 decoupled deployment and release, 282
Constant builds, 132 delivery pipeline, 24, 33, 41, 46, 128, 200,
constraints, 33, 35, 36, 37, 46, 123, 214, 214, 238, 244, 249, 271, 273, 274, 278,
235 279
Continuous Delivery, 15 Deming, 7, 17, 18, 21, 22, 28, 33, 34, 35,
Continuous Delivery/Deployment, 62 53, 54, 64, 66, 68, 69, 79, 87, 102, 137,
Continuous Deployment, 261 143, 150, 160, 185, 190, 222, 245
Continuous Improvement, 82, 201, 256, Deming’s Deadly Diseases of
262 Management, 7, 53
continuous improvement culture, 87, 166 Denialism, 10, 161, 162
continuous integration, 25 deployment pipeline, 128, 129, 131, 141
Continuous Integration, 15, 62, 72, 73, Deployment script, 131
74, 97, 100, 129, 178, 179, 186, 228, Deployment time, 273
239, 240, 247, 251, 259, 267, 270, 273, Detecting silos, 227
275, 280 DevOps Business Advantages, 180
Continuous Learning Organization, 262 DevOps Communication Roles, 9, 146
266
DevOps Engineer, 285 Everyone is Responsible for Quality, 224
DevOps in a box, 206 Explicit and Implicit Coercion, 156
DevOps Metrics, 272
DevOps pipeline, 90, 95, 96, 98, 117, 149, F
171 Facebook, 30
DevOps Release Strategies, 281 Facilitation and Support, 155
Differing Perceptions, 153 factory floor, 111, 270
Disconnect Personal Gains from fail fast and learn from it, 103, 136, 257,
Organizational Objectives, 217 263
distributed version control system, 90 failure proposition card, 174
Dojo’s, 22, 167, 276 False Prophets, 285
Dojos and Centers of Excellence, 276 fashion show victim, 257
Don’t Give Up, 220 favoring a plan over planning, 165
done criteria, 114 fear of reprisal, 137
double, 18, 212, 263 Fear of the Unknown, 152
Double Loop learning, 212 feature creep, 108
downstream, 46, 62, 183, 185, 198 Feature Toggles, 283
Dr. Edwards Deming, 14, 190 features, stories, 128
Dr. Edwards W. Deming, 16, 17 Feedback Loop, 83
Drift, 90, 109, 273, 278 feedback loops, 8, 82, 108, 132, 137, 181,
dysfunctional software, 126 186, 201, 221, 230, 245, 263, 274
Feedback Loops, 11, 54, 82, 234, 245,
E 271
Eaton Corporation, 38, 39 Finance, 19
Educate, 219 finger pointing, 42, 136, 227
Education and Communication, 155 Five Focusing Steps, 11, 213, 214
Eliminate Waste, 235 flexible and reactive systems evolution,
Emphasis on Titles, 66 282
Enable developers, 278 Flipping the Testing Pyramid, 8, 98
Encapsulation, 236 Focusing on CI, 264
Entity Attraction, 143 Followers, 58
epics, 128, 129 Forcefield Analysis, 10, 154, 155
Escalation of Commitment, 142 full stack engineers, 27
evangelism, 258 Functional Attraction, 142
Evangelize, 11, 216 Functional automation, 100
evangelizing, 102, 105, 216, 217, 219 Functional Automation, 74
267
G Index, 292
informal channels of communication, 147
Gatekeeper, 146
Informal Organizational Culture, 10, 156
gatekeeper of quality, 224
infosec, 277
GE, 166, 167
Infrastructure code security, 278
General Electrics, 165
Innovative Change, 151
General Mistrust, 152
innovators, 102, 140, 146, 203
Getting the CI habit, 263
Innovators, 58
Git, 90, 237
Instrumentation & Control, 201
GIT, 90, 266
intervention techniques, 175
GitHub, 238
Inverse Conway Maneuver, 126
Goldratt’s Theory of Constraints, 35, 36
IT as Core Business Component, 10, 196
governance, 277, 278
IT is the new Factory Floor, 11, 198
grapevine, 156
It’s about Culture, 187
Guidance by a Change Agent, 150
iterating, 113
H iteration, 84, 85, 92, 105, 113, 115, 121,
124, 224, 228, 239, 245, 246, 247, 259,
Hero Cult, 65, 152
267, 268, 274, 279
hero cult tendencies, 103
ITIL, 15
Heroes, 157
ITSM, 15
Hidden Constraints, 9, 122, 123
Hidden Persuasion, 156 J
hidden’ bottleneck, 123
Higher Acceptance Rate, 235 Jack Welch, 166
Jargon, 145
Horizontal Optimized, 45
Jenkins, 97
How DevOps Helps Break Silos, 227
Jidoka, 39
How often do you Deploy, 272
Jonny Leroy, 126
How to Deal with Negative Politics, 9, 147
Just-in-Time, 40
I
K
Improved reputation, 215
Kaizen, 8, 40, 82, 107, 201, 270
Improved workplace synergy:, 214
Kanban, 36, 40, 55, 56, 174
Improvement is Learning, 266
Kanban boards, 36
incentive structure, 27, 46, 102, 103, 202
Incentivizing, 26 L
inception phase, 108
Laggards, 57, 58, 59, 153
268
Language, 157 material and information flow mapping,
Lean, 7, 15, 19, 22, 25, 30, 38, 39, 64, 66, 206
69, 72, 82, 87, 107, 111, 123, 137, 138, Matrix forms, 195
139, 143, 150, 166, 185, 190, 197, 198, Matt Simons, 126
199, 201, 208, 222, 245, 269, 272, 286 Maven lifecycle, 100
Lean Manufacturing, 22, 25, 38, 39, 111, MBA, 65, 159, 160, 190
139, 190, 197, 198, 199, 201, 286 MBO, 68, 69
Lean Manufacturing revolution, 22, 38, Mean time to restore service (MTTRS), 274
39, 190, 286 Meetups, 58
learning organization, 22 merge conflicts, 91
legacy databases, 86 M-form organization, 195
life span of a product, 174 micro services,, 200
LinkedIn, 30, 279 Monitor Configuration file, 131
Listening, 10, 164 monitoring, 20, 21, 61, 108, 132, 138,
living documentation, 117 186, 201, 222, 228, 232, 233, 239, 245,
load/stress testing, 100 246, 259, 264, 267, 275, 279
Logging, 231 motivated reasoning, 162
logical gates, 252 Move to Modularization, 199
Long Range Orientation, 150 Moving from a Ticket Taking Culture, 110
Loose coupling, 95 MTTR, 272
Loss Aversion Bias, 141 MTTR Improvement, 278
Loss of, 153 muda, 208
Murphy’s Curve, 11, 209
M Must Earn Trust, 217
mainline, 237, 283 mutation testing, 238, 243
maintenance cost, 243 Mutation testing, 133
Manage Informal Coalitions and Cliques, Myths, 157
148
N
Management by Objectives, 7, 67, 68
Managements Role in DevOps, 255 national culture, 158
manual build process, 108 negative politics, 147, 148
manual test phase, 97 Netflix, 271
manual testers, 108 N-form organization, 195
Marketing, 19, 28, 30, 193 Non value adding time, 208
Masters of Business Administration, 64 non-value adding times, 208
norms, 148, 157, 158
269
O political players, 147
Poor Timing, 153
offshoring, 30, 42, 43
Prepare for Confrontations, 219
One size rarely fits all., 258
pride of workmanship, 26
Opinion Leaders, 146
principles of Agile, 112
opportunities, 102, 144, 170, 171, 212,
Process Flow Diagrams, 207
266
product owner, 113, 114, 115, 117, 235
Orchestration Versus Automation, 11,
product owners, 59, 235
249
productivity, 17, 21, 25, 38, 138, 182,
Organization Factors - Spatial
199, 214
Arrangements, 144
project managers, 90
Organizational Ability, 263
Protocols, 131
Organizational Behavior, 138, 139, 141,
Protractor, 199, 241
142, 187, 276
Puppet, 199
Organizational Development, 9, 126, 149,
push on green, 261
150, 154, 175, 176
Organizational Structures, 10, 191 Q
Out of the Crisis, 17
Overcoming Forces of Resistance, 10, 155 quality assurance, 20, 22, 24, 30, 42, 49,
73, 81, 204
P Quality Assurance, 24, 71, 74, 83, 91, 97,
100, 101, 103, 118, 136, 145, 188, 194,
Page Objects, 242
212, 221, 224, 225, 242
painting by the numbers, 258
Participation and Involvement, 155 Quality Control, 225
Quality Gate Detection, 274
PDCA, 15, 21, 201, 245
quality gatekeeper, 101
Peer Pressure, 153
Quality Gates, 11, 229, 239, 247, 251,
performance testing, 66, 100, 124, 243
252, 259, 267, 275, 280
Personality Conflict, 153
Quality Profiles, 251
Peter Drucker, 67
Quality verification time, 273
pilot project, 43, 50, 59, 206
Pilot Releases, 282 R
Pipeline lockdown, 278
Pipeline success rate, 273 ramp-up period, 210
Pipeline Swarming, 243 randomizing, 253
Pipeline up time, 273 record and play automation, 286
Plan, Do Check, Act, 15 Reduce the Uncertainty, 148
Planned Change, 149 Reduced operational costs, 214
270
Reduction in Inventory, 214 Six Sigma, 15
Release Notes, 131 skills, 15, 20, 27, 64, 66, 102, 165, 189,
Revolutionary Change, 151 194, 204, 243, 285
Risk reduction, 132 Social Disruption, 153
Rites, 157 social environment, 276
Rituals, 157 Social Environment, 142
ROI, 18, 20, 48, 109, 174 Social Support, 143
Role-Organization Factors, 145 Social-Psychological Factors, 149
Role-Set Factors, 144 SonarQube, 48, 95, 173, 235, 251, 270
rollback, 132, 136 Source Code Management, 89, 97
Ronald Fisher, 253 Sources of Change Agent Resistance, 10,
152
S Sources of Resistance to the Change
Sales, 19, 28 Agent, 9, 151
scapegoats, 147 Specification by Example, 8, 91, 114,
scientific approach, 246 116, 117, 118, 130, 145, 163, 228, 235,
SCM, 8, 85, 89, 90, 91, 92, 106, 115, 121, 239, 242, 246, 259, 267, 270, 274, 279
124, 278 Spreading the word, 264
SCM Strategy, 89 Sprint Review, 85, 92, 106, 115, 229,
Scrum master, 23 239, 247, 259, 268, 275, 280
Scrum Master, 18 Sprints, 71, 76, 84
Security and Compliance, 277 Standardize infrastructure, 200
Security testing, 100, 278 Standardized Parts, 199
Selenium, 100, 199, 241, 242 statistical sample subset, 282
Self Interest, 152 statistical sampling, 281
Set, 144, 147 stifle innovation, 136
Shadow Responsibilities, 66 Stories, 71, 106, 115, 121, 124, 157
shifting testing left, 18, 20 strengths, 171
silos, 8, 19, 24, 25, 33, 80, 81, 103, 109, Striving for Green, 261
114, 126, 153, 185, 186, 195, 202, 221, subject matter experts, 276
226, 227 Superordinate Goals, 9, 148
single loop learning, 18 Superstars, 157
Single Loop learning, 212 Survey Feedback, 10, 175
Single versus Double Loop Learning, 11, SWOT, 10, 168, 170, 171, 172, 177
211 SWOT Analysis, 170, 171
situational analysis, 171 Symbols, 157
271
Symptoms of Stress, 9, 143 The process is the documentation, 132
System of Profound Knowledge, 7, 31, 33, themes, 128, 129, 178, 182, 183
35 Theory of Constraints, 11, 35, 46, 68,
123, 213, 214
T Those without Character Will Fail, 218
Tactlessness, 153 ThoughtWorks, 126
Takt Time, 40 threats, 156, 170, 171, 277
Target, 22, 142, 167, 276 Threats to Rational Decision Making, 9,
TDD, 11, 74, 228, 234, 235, 236, 239, 141
246, 259, 267, 274, 275, 279, 280 Three Categories of Change, 9, 151
Technical Debt, 7, 47, 48, 95, 173 ticket taking culture, 19
telemetry, 18, 20, 21, 201, 228, 239, 240, tight code coupling, 92
246, 247, 259, 260, 267, 268, 275, 279, tightly coupled code, 94, 95
283 Tightly Coupled Code Issues, 8, 94
Telemetry and Monitoring, 230 TOC, 35, 214, 215
test after the fact, 20 Top-management support, 256
Test data, 130 Toyota Production System, 21, 39, 139,
Test Driven Development, 57, 74, 79, 234, 206, 269, 270
242, 245 TPS, 21, 39, 269
Test run, 130 TQM, 15
testing, 18, 20, 21, 22, 23, 25, 30, 33, 53,
U
54, 57, 66, 72, 73, 74, 78, 79, 81, 83,
85, 91, 95, 97, 99, 100, 101, 103, 104, U-form organization, 195
105, 108, 109, 114, 115, 117, 118, 121, Understand Other’s Perspectives, 219
123, 124, 126, 127, 129, 130, 132, 133, Understanding the Social Environment,
134, 145, 150, 178, 187, 188, 199, 204, 9, 142
214, 222, 224, 225, 234, 238, 241, 242, uniqueness, 163
243, 252, 253, 259, 261, 270, 273, 278, Unit testing, 57, 73, 76, 78, 79, 95, 97,
284, 285 99, 100, 104, 124, 150, 163, 187, 188,
The Elements of Organizational Culture, 234, 243
10, 157 upstream, 62, 183, 185, 198
The Envelope Game, 8, 119 User Acceptance Testing, 33
The Father of the Quality, 17 User Stories, 71, 84, 92, 105, 106, 115,
The Fear of Failure, 152 121, 124, 228, 239, 246, 259, 267, 274,
The Mirror Image Fallacy, 9, 139 279
The Practice of Management, 68
272
V
value adding times, 208
Value Proposition Cards, 10, 173, 174
value stream, 107, 108, 109, 123, 207,
208
Value Stream Map, 123
Value Stream Mapping, 11, 33, 123, 205,
206, 207, 208, 217
Value Stream Metrics, 208
value streams, 107, 108, 109, 207
values, 28, 143, 157, 158, 212, 219, 265
variances, 137
Version Control, 33
Version management, 128
Versioning, 9, 128
Vertical Optimized, 45
Vision, 102, 144
W
Walking the talk, 265
weaknesses, 171, 232
Whole Foods, 17
Wikis, 91
WIP,, 272
Word documents, 91
work-out approach, 166
273

A Tale of Two Companies

Hochgeladen von

Dokumentinformationen

Copyright

Verfügbare Formate

Dieses Dokument teilen

Dokument teilen oder einbetten

Freigabeoptionen

Stufen Sie dieses Dokument als nützlich ein?

Sind diese Inhalte unangemessen?

Copyright:

Verfügbare Formate

A Tale of Two Companies

Hochgeladen von

Copyright:

Verfügbare Formate

A Tale of Two Companies

A Narrative Story Based

Copyright © Thomas Gilmore

Tom is an Agile/DevOps Transformation Coach. Tom brings a unique

With over 20 years of experience in Lean Manufacturing, TQM, Agile

This book is an attempt to help explain why so many organizations have

This book is intended for anyone, technical or non-technical who would

Let’s Summarize what we know and have learned: .................................................................. 15

Learning Point(s): .................................................................................................................................................................................................................................................. 15

The Journeys Begin.......................................................................................................................................................................................................................................................... 33

Breakdown of Orca’s Problems: ............................................................................................................. 35

Learning Point(s): .................................................................................................................................................................................................................................................. 35

Orca Evaluates Their Choices ................................................................................................................................................................................................................................... 44

Breakdown of Orca’s Choices: ................................................................................................................. 45

Learning Point(s): .................................................................................................................................................................................................................................................. 46

Orca Makes a choice........................................................................................................................................................................................................................................................ 49

Breakdown of Orca’s Choice: ................................................................................................................... 51

Learning Point(s): .................................................................................................................................................................................................................................................. 52

The Implementation begins ........................................................................................................................................................................................................................................ 58

Breakdown of Orca’s Implementation: .............................................................................................. 60

Changing the Culture ........................................................................................................................................................................................................................................ 61

Orca Must Make Tough Decisions ......................................................................................................................................................................................................................... 68

Breakdown of Orca’s Dilemma: .............................................................................................................. 71

Learning Point(s): .................................................................................................................................................................................................................................................. 71

Orca and Dolphin’s Project ......................................................................................................................................................................................................................................... 77

Breakdown of Iteration 1: ........................................................................................................................... 78

Learning Point(s): .................................................................................................................................................................................................................................................. 79

Learning Point(s): .................................................................................................................................................................................................................................................. 84

Breakdown of Iteration 3: ........................................................................................................................... 95

Learning Point(s): .................................................................................................................................................................................................................................................. 95

Iteration 4............................................................................................................................................................................................................................................................................. 103

Breakdown of Iteration 4: ........................................................................................................................ 103

Learning Point(s): ............................................................................................................................................................................................................................................... 104

Shift Left and Continuous Testing ...................................................................................................................................................................................................................... 107

The Envelope Game ..................................................................................................................................... 109

Iteration 5............................................................................................................................................................................................................................................................................. 111

Breakdown of Iteration 5: ........................................................................................................................ 111

Learning Point(s): ............................................................................................................................................................................................................................................... 112

Iteration 6............................................................................................................................................................................................................................................................................. 114

Breakdown of Iteration 6: ........................................................................................................................ 114

Learning Point: .................................................................................................................................................................................................................................................... 115

Iteration 7............................................................................................................................................................................................................................................................................. 116

Breakdown of Iteration 7: ........................................................................................................................ 117

Learning Point(s): ............................................................................................................................................................................................................................................... 117

Iteration 8............................................................................................................................................................................................................................................................................. 123

Breakdown of Iteration 8: ........................................................................................................................ 123

Learning Point(s): ............................................................................................................................................................................................................................................... 124

Creating Culture Change through Organizational Behavior............................................................................................................................................................. 127

What we will learn ........................................................................................................................................ 128

The Mirror Image Fallacy ............................................................................................................................................................................................................................. 128

Let’s take a look at Dolphin now .......................................................................................................................................................................................................................... 144

Breakdown of Dolphin’s Problems: ................................................................................................... 145

Learning Point(s): ............................................................................................................................................................................................................................................... 146

Dolphin Looks for Answers ...................................................................................................................................................................................................................................... 152

Breakdown of Dolphin’s Answers: ..................................................................................................... 153

Learning Point(s): ............................................................................................................................................................................................................................................... 154

Dolphin Reviews the Evidence ............................................................................................................................................................................................................................... 159

Breakdown of Evidence Review: .......................................................................................................... 161

Learning Point(s): ............................................................................................................................................................................................................................................... 162

Dolphin Learns about Agile and DevOps ....................................................................................................................................................................................................... 167

Breakdown of what Dolphin Learned: ............................................................................................ 169

Learning Point(s): ............................................................................................................................................................................................................................................... 169

Dolphins Ah Ha Moment ........................................................................................................................................................................................................................................... 173

Breakdown of the Ah Ha Moment: .................................................................................................... 174