Equipping PMI Members with Agile Skills and Knowledge

PMI Agile Community of Practice

The Agile Community of Practice focuses on delivering knowledge and providing a virtual networking forum for agile practitioners. This includes anyone interested in, working in, or impacted by developments in the collection of good practices, principles and techniques in agile project management.

The Community
Enables agile practitioners to contribute and share their knowledge with the global PMI community. Explores the relationship between agile principles and practices. Communicates how agile may differ from, or complement the teachings within, PMI's A Guide to the Project Management Body of Knowledge (PMBOK Guide).

Experience Report Abstract

This experience report covers a systems engineering team from early 2008 to Mid-2009 as the team implemented Scrum and then Kanban. This experience report is meant to provide some insight into real world successes (and challenges) of these practices. This report also illustrates some of the challenges and opportunities of systems engineering teams versus development teams. Finally, the report demonstrates some of the challenges of managing multiple small projects with a single team of resources. The author of this report served as manager of the systems engineering team. The observations are his and do not reflect those of his company. Key Findings: Mangers and technical professionals are increasingly being placed in the role of project manager for small and mid-sized projects. A key aspect of these projects in balancing project and functional work demands. Kanban and Scrum are applicable to environments where a significant percentage of the work is un-planned (e.g., production support). Practitioners need to understand the forces behind various techniques in order to best match the techniques to the type of work and the ability of the organization to successfully use the techniques Providing visibility into the work and the prioritization process both reduced the frequency of expedited work and improved overall throughput.

Please do visit the PMI Agile Community of Practice site http://agile.vc.pmi.org to raise questions and discuss this report.

Experience Report: Applying Kanban and Scrum in a Systems Engineering Environment Sellers Smith 2011 All rights reserved

Equipping PMI Members with Agile Skills and Knowledge

Experience Report
Background
This experience report covers R&D work in the email marketing division of mid-sized multinational corporation. The division sells email services (Software as a Service) which are delivered via Internet access to products. At the time of the experience report, the divisions product line consisted of: Two legacy products, in production, but not being enhanced One product in production and actively being enhanced Two new products that were developed and launched during the time period of this report.

Customers for these services are mid-sized and large commercial companies located around the Globe - North America, Europe, Asia-Pacific. The various products provide the ability to send and track emails from single emails with digitally secured attachments to multi-million recipient campaigns with customized/personalized templates. Two of the products have web-based UI, while three of the products have programmatic interfaces (i.e., SOAP and REST). The various products have some significant non-functional requirements: Highly available for global customer base Highly Scalable due the periodic nature of large volume marketing campaigns and the need to support time sensitive personal emails Secure (for emails sending digitally secured attachments) Low cost of operations

Organization
The R&D organization had four development teams (ranging from two to six developers), a data warehouse and database administration team, one QA team and the system engineering team(s). The development teams were organized around products under active development, and QA and system engineering resources were matrixed into the teams. Each product team had separate release schedules. Each development team did four week Sprints and after the initial release, each Sprint ended in a release. The system engineering team was responsible for several distinct types of projects and other work: Building and upgrade infrastructure to support development and production. Supporting software development projects Monitoring the operational environments and responding to support issues

Experience Report: Applying Kanban and Scrum in a Systems Engineering Environment Sellers Smith 2011 All rights reserved

Equipping PMI Members with Agile Skills and Knowledge The systems engineering team had two sub-teams - Blue (six members in the US) and Gold (four members in India). All members of the team were specialized generalists with strong linux system administration skills and also specialized in one or more disciplines and or technologies - e.g., Networking, mySQL database administration, monitoring. The three existing products were developed on local servers, and deployed into a company owned, but geographically remote Data Center with external resources managing the server and networks (e.g. ping, power, pipe). The new products developed during this experience report were developed and deployed into a private cloud - virtualized environments operated by a third party.

Challenges
At the start of this experience report, the systems engineering team faced numerous challenges. The team was responsible for a wide variety of work including: Meeting non-functional requirements (e.g., scalability, availability, security), Performing infrastructure improvements and upgrades, Supporting various development teams - from setting up development environments to deploying applications to production, Systems engineering activities including configuring servers, installing 3rd party software, building monitors, data back-ups, Providing 24x7 support to the platform by responding to alerts, resolving production incidents and conducting analysis of production incidents

The released quality of the existing products was not good. There were daily/weekly production defects and/or performance problems. These production issues required time from the system engineering team to diagnose, resolve and perform root cause analysis. Additionally, the issues resulted in demands to implement new monitoring and make changes to production configurations. Deployment of new releases to production needed to be done by the development team. These deployments were time consuming (taking multiple hours), error prone and frequently required post-release patches. Additionally, these production changes required that systems engineering co-ordinate with multiple parties (e.g., development, networking, server administration) including remote and external parties.

Process Improvement
Move to Scrum
The move to Scrum started not with a desire to get more agile, but an attempt to make sense out of the large and varied activities flowing into the systems engineering team. The first step was to set a vision product vision for the organization. While distinctly different from the typical software product vision, the vision allowed the team to organize the wide variety of work and competing demands. The vision was:

Experience Report: Applying Kanban and Scrum in a Systems Engineering Environment Sellers Smith 2011 All rights reserved

Equipping PMI Members with Agile Skills and Knowledge Keep it Running. Top priority was to resolve any production and or client impacting incidents. Production incidents were resolved immediately, and did not make it into the backlog. Make it Better. Second priority went to activities supporting projects to build or enhance the products - both application enhancements and infrastructure upgrades. Reduce Operating Cost. Third priority was to activities that would reduce the amount of time and effort required to operate the platforms.

All of the existing work was collected into a backlog. The backlog was prioritized by the VP of Operations, acting as a product owner for the various development teams. VersionOne was in use by the development teams and so was adopted by the systems engineering team. One systems engineer was assigned as point person for each development team and or infrastructure project. This point person was responsible for entering and tracking work required by the various teams and projects, effectively acting as a project lead for each project. When the systems engineering team needed an activity performed by an external party (e.g., vendor, networking, implementation engineer), the activity was assigned to the systems engineer coordinating the work, and the effort estimate reflected the effort of a systems engineer to coordinate the work. While there were still scheduling conflicts between the various teams and projects, this system quickly surfaced resource bottlenecks - which were resolved by the VP of Operations. The various project leads were able to mitigate and communicate any impacts to the various project stakeholders. The Blue (US Team) held a weekly planning meeting to review work completed in the past week and to plan work for the up-coming week. Work was split between the Gold and Blue teams, with a member of the Blue Team, representing the Gold Team. There were a pair of daily-stand-ups, one for the Blue and one for the Gold team. The Gold team stand-up was by phone. The definition of 'done' varied by the type of work. Some work was done when completed in or deployed to production. Larger pieces of work (stories) might be split into two or three parts - design, build and or deploy - if there was either a significant amount of work, or calendar time involved in the steps. For example, a server upgrade might be broken into design/build and deploy. This definition is counter the agile concept that each story must deliver User Value and Deployable Software. There was a lot of discussion and experimentation around incorporating work on production incidents into the backlog of work. Generally, production incidents were not counted in velocity or added to the backlog. However, follow-up work that exceeded 4 hours (e.g., root-cause analysis, better monitoring, updating system documentation) was added to the backlog. Generally, the team proceeded from a couple of assumptions. First, production incidents had to be resolved and would be within a few hours. So there was no benefit in

Experience Report: Applying Kanban and Scrum in a Systems Engineering Environment Sellers Smith 2011 All rights reserved

Equipping PMI Members with Agile Skills and Knowledge adding them to the backlog, or tracking them. Second, SEs were scheduled to be on-call and working on production incidents. So, the expectation was that this time was not available to contribute to working backlog items. While things were clearly better and more organized with Scrum elements, there were a number of issues. First, new priorities were emerging due to production incidents and problems and customers (e.g., VP of Operations) were unwilling/unable to wait until the weekly prioritization meeting. A significant number of items were not getting completed during the 1 week sprint due to external dependencies. Third, production incidents tended to fluctuate the amount of time available and work completed by the team.

Move to Kanban
Like the move to Scrum, the move to Kanban, was done in response to various problems. First, the team began allowing (acknowledging) the addition of hot items on a daily basis. As a consequence, the weekly planning/review meeting was dropped in favor of pulling items on a daily basis. Soft Work-in-Progress (WIP) limits were set. Each team - Blue and Gold - attempted to maintain between 2 and 2.5 items in-progress per team member. Team members would generally pull a new item if they had only a single item. The Kanban Board was an adaption of the Scrum Board (or rather VersionOne). Gold and Blue teams each maintained a single board of work in progress, items were either available, in-progress, completed or done. The team retained velocity as a metric, rather than attempting to calculate and improve on cycle time. Basically, points were accumulated for work in the week (formerly Sprint) in which the backlog item was completed. This proved adequate to allow forecasting of capacity and when work would be completed. A monthly planning meeting was set to prioritize items in an 'available' queue for each month. This allowed the team to focus on work that was immediate, and avoided the need to prioritize or re-prioritize work that would not be done for one or more months. The team did adopt and find value in some of the Kanban techniques. However, Kanban was really limited to a single team. Thus, the team did not do value mapping to identify bottlenecks and improve the overall flow.

Technical Improvements
In addition to the process improvements, the team introduced a number of technical improvements to facilitate their work. The most valuable (and time consuming) effort was to build staging environments that replicated the production environment as closely as possible. The work involved in building
Experience Report: Applying Kanban and Scrum in a Systems Engineering Environment Sellers Smith 2011 All rights reserved

Equipping PMI Members with Agile Skills and Knowledge and synchronizing production and staging led to a much better understanding of the production configuration and uncovered a number of issues and opportunities for improvement. Once built, the staging environment was available to dry-run software deployments, monitor and other systems engineering work, prior to moving to production. Another valuable technical improvement was to introduce Splunk - a tool to capture, index and query log files. These index log files were available for troubleshooting and root cause analysis. Additionally, reports from the logs were reviewed with development on a weekly basis and led to the identification and resolution of numerous minor production issues. Over the course of this experience report, both development and production environments moved heavily to virtualization or 'cloud' computing. The two new products were developed and deployed entirely in private clouds. The development environment for other legacy products was moved to a VMWare environment. The systems engineering team found a significant productivity boost in the ability to define and re-use virtualized environments without worrying about hardware configurations.

Lessons Learned
Process improvement is a journey, not a destination. The true value is in getting better every day rather than implementing any methodology even Scrum or Kanban. The team did not focus on implementing Scrum or Kanban, but rather on adjusting its processes to address its most pressing problems and exploit opportunities to improve. Key to the teams ability to successfully mix and adapt techniques from Scrum and Kanban was understanding the pre-requisites and purpose of the various techniques. Techniques that are critical in one context may be redundant or counter-productive in another. The team needed to know when to work on correctly implementing one technique and when to drop, adapt, or defer another.

Conclusion
The result of the various process and technical improvements was a significant increase in software availability and capacity. Additionally, production incidents and number of alerts were cut by over 50%, with the the number of false alerts cut even more significantly. In addition, the amount of effort to respond to production issues was cut from around 4 FTE to less than 2 FTE. The team also experienced significant increases in morale, as they were able to focus on making the system better and their job easier, rather than responding to production issues. This team's adoption of Scrum and Kanban techniques was not picture perfect and some would argue that what the team did was not true Scrum or Kanban. Rather, the team adapted and adjusted in response to current issues. However, key to this adaption and adjustment was an understanding of the rationale behind the various processes and practices. The team kept practices that were working and adapted or dropped those that
Experience Report: Applying Kanban and Scrum in a Systems Engineering Environment Sellers Smith 2011 All rights reserved

Equipping PMI Members with Agile Skills and Knowledge were not. Additionally, the team adjusted the process to deal with current issues and to explore opportunities for improvement.

Experience Report: Applying Kanban and Scrum in a Systems Engineering Environment Sellers Smith 2011 All rights reserved

Equipping PMI Members with Agile Skills and Knowledge

About the Author

Robert Sellers Smith. Sellers has spent his professional career building software systems in one capacity or another. His projects range from small development efforts where he was the sole contributor to multi-year, 100+ person Federal Programs. Sellers has served in a variety of roles project manager, engineer, architect, and tester. For the past ten years, he has concentrated on building internet applications and software as a service. He is currently Director of Quality Assurance and Agile Evangelist at Silverpop.

Copyright
This material has been reproduced with the permission of the copyright owner. Unauthorized publication of this material is strictly prohibited. For permission to re-produce this material, please contact any listed author.

Experience Report: Applying Kanban and Scrum in a Systems Engineering Environment Sellers Smith 2011 All rights reserved