March 31 – April 2, 2009

The Marriage of
Agile and Test
Management
Brian Massey, Sr. Product Manager, IBM
 Who Am I

ƒ Product Manager
ƒ No worries – not here to sale you anything (but to ensure the kids
get fed I have to mention them)

ƒ As part of Rational Services organization helped develop and deploy

frameworks for our customers

ƒ Established Software Quality Centers of Excellence for Companies

ƒ Experience as Developer, Tester, Architect, Dev Manager, QA Manager

ƒ Used to build Air Defense Systems for a living

ƒ Developed and tested embedded systems

ƒ Father of 6 so prepared for anything

 Agenda

ƒAgile Software Development

ƒ “Traditional” Software Testing

ƒ Everything is different but still the same

ƒ The fight is on

ƒ IBM Agile Project

ƒ Bringing it all together

3
The Marriage
 5
Agile Defined !
ƒ Agile software development refers to a group of
software development methodologies that promotes
development iterations, open collaboration, and
process adaptability throughout the life-cycle of the
project.
ƒ Things are done in small increments, with minimal
planning, rather than plan at length.
ƒ Project adapts to changes
ƒ Emphasis on stakeholder involvement.
Agile Principles 6
ƒ Customer satisfaction by rapid, continuous delivery of
useful software
ƒ Working software is delivered frequently (weeks rather
than months)
ƒ Working software is the principal measure of progress
ƒ Even late changes in requirements are welcomed
ƒ Close, daily cooperation between business people and developers
ƒ Face-to-face conversation is the best form of communication (Co-
location)
ƒ Projects are built around motivated individuals, who should be trusted
ƒ Continuous attention to technical excellence and good design
ƒ Simplicity
ƒ Self-organizing teams
ƒ Regular adaptation to changing circumstances
 Agile -- IBM/Rational System Test Team? 7
7

ƒ Agile is the ability to react to change

quickly and use customer feedback to
impact the current release. It also allows
delivery of iterations to downstream
teams for testing and further
development through frequent delivery
of working software.

ƒ Agile is continuous testing and validation

with constant collaboration between
teams. It is not an excuse to abandon all
documentation and process.

ƒ Not all teams can and will use all aspects

of agile. Many need to make transitions
more slowly and use a modified
approach.
Agile Planning 8
 9
Test First Development (TFD)

ƒ Automated acceptance testing drives your

detailed requirements efforts
ƒ Automated developer testing drives the detailed
design of the system
ƒ Tests form the majority of your detailed
specifications
ƒ You’ll still need some high-level models to
provide overviews
ƒ This is the first time in IT history that developers
are motivated to keep the specifications up-to-
date
ƒ An example of single sourcing information

9
Test Driven Development (TDD) 10

ƒ Instead of code driving the testcases; use

cases/testcases define the features and code that
are delivered for a milestone or iteration

ƒ Development teams deliver a use case per feature or

a use case that covers multiple features before any
coding is done

ƒ A good use case contains the following:

ƒ Overview – Brief description of what will be delivered
ƒ Any known pre-conditions
ƒ High level steps
ƒ Expected results

ƒ Once the use cases are complete then coding begins

ƒ Teams may choose to provide viewlets of expected

behavior over use cases
ƒ Viewlets provide a mock up of the feature with a
developer taking you through the expected behavior
 Agenda

ƒ Agile Software Development

ƒ“Traditional” Software Testing

ƒ Everything is different but still the same

ƒ The fight is on

ƒ IBM Agile Project

ƒ Bringing it all together

11
 12
Development Lifecycle Methodologies
Generic “V” cycle

12
 Agenda

ƒ Agile Software Development

ƒ “Traditional” Software Testing

ƒEverything is different but still the same

ƒ The fight is on

ƒ IBM Agile Project

ƒ Bringing it all together

13
 14
Business Basics
ƒ Test Early, Test Often

ƒ Use Iterative Approach

ƒ Move Setup and Overhead into Earliest Iteration

ƒ Minimize Documentation, Maximize Testing

ƒ Have defined objectives to assess against (scope)

Points to Ponder 15

ƒ What is your test project Vision?

ƒ What are your deliverables?

ƒ What are your goals?

ƒ How do you know when those have been met?

ƒ Have defined objectives to assess against (scope)?

ƒ Are you effectively utilizing your resources (people, lab, etc)?

Why do we test? 16

ƒ Find Bugs?

ƒ Validate Specs?

ƒ To break things?

ƒ To get developer in trouble?

ƒ Ensure system is ready to use!

ƒ To create release notes…

Expense of Software Testing 17

ƒ Testing is Expensive…. But…

ƒ What is cost of support?
ƒ What is cost of a Crit Sit?
ƒ What is cost of business due to poor quality?

ƒ When say testing expensive – what is that compared to?

 Agenda

ƒ Agile Software Development

ƒ “Traditional” Software Testing

ƒ Everything is different but still the same

ƒThe fight is on
ƒ IBM Agile Project

ƒ Bringing it all together

18
 19
Agile Testing vs. Traditional Testing
www.ambysoft.com/essays/whyAgileWorksFeedback.htm

19
The Extremes 20
 21
Approaches to Quality

• Linear, Sequential
• QA planning occurs late
• No test case review
• Emphasis on bug fixing
Continuous Testing in an 22

Agile Environment

User Stories \ Code

Code Feat.
Accept. Tests \ Unit Test Handoff to
Feature WOs
Tasks QA
Dev Signoff
Design Handover

Test Case Review

Release
Definition
(RDD / PRD)

QA Signoff
QA Plan \ QA Feature
Author Feature Feat. System Product
Automation Test
Test Cases WOs Test Release
Plan Execution
 Agenda

ƒ Agile Software Development

ƒ “Traditional” Software Testing

ƒ Everything is different but still the same

ƒ The fight is on

ƒIBM Agile Project

ƒ Bringing it all together

23
IBM SVT Agile practices 24

ƒ Fully completed use cases within an iteration

ƒ Active user involvement/continuous stakeholder feedback
ƒ Adaptable plans and goals with self directing teams
ƒ Stable code at iteration end
ƒ Reflections/Post mortems for continuous process improvement
ƒ Automate Unit and Functional Testing, System where appropriate
ƒ Continuous builds and integration
ƒ Continually verify quality
ƒ Daily interactions between teams and business people
ƒ Customer demos at each iteration
ƒ Develop iteratively with timeboxed functioning milestones
ƒ Prioritize features based on customer feedback, eliminate waste
ƒ Modeling and prototyping
ƒ Test driven development
ƒ Document templates and iteration assessments
 25
Case Study - RPT
ƒ Problem:
ƒ Large amount of dependencies on underlying code
ƒ Global team with 2 locations, smaller than RAD team
ƒ Small code base but large scale test requirements
ƒ Smaller 4 week iterations that never truly functioned

ƒ Solution:
ƒ Constant Development and Test collaboration
ƒ Daily SCRUM meetings
ƒ Better use case development
ƒ Internal customer adoption as well as external deliverables and stakeholders

ƒ Tools Used:
ƒ ReqPro for tracking integrations and requirements
ƒ CQ for defect management
ƒ RFT and RPT for automation
ƒ RMT and CQTM for manual testing
ƒ Reuse of artifacts from RAD and the RAM environment as a system under test

25
 26
More about testing iterations
ƒ Establish function and system testing goals for iterations
ƒ When do you plan to run your performance benchmarks? Scalability runs? Identify those
major areas of focus for each iteration including documentation, install, and
comsumability.
ƒ Target regression testing of various areas across iterations. Ensure that previously released
function is working.

ƒ Release an assessment report upon completion of any iteration or milestone test

ƒ Did the teams meet the goals of the iteration? Did we have any subjective feedback on
usability, consumability etc. input from customer groups?
ƒ Are we making our performance and scalability goals?
ƒ Are all blocking or high severity defects addressed?
ƒ Is there an agreed-to plan for fixing the remaining unresolved defects?
ƒ How are we handling customer feedback?

ƒ Test case development becomes iterative

ƒ Think of the iterations as “mini-releases” and plan function and system-level tests
accordingly.
ƒ Enhance/update the testcases across iterations to make them more complex (and
complete) as additional features are introduced.

ƒ Higher risk items should be tested as soon as they are delivered

ƒ If performance is a key item, it should be tested early when there is time to fix problems.
Performance tests that require longer to run will overlap multiple iterations. Defects found
must be prioritized with the other work (If defects do not get fixed, then the value of early
testing is reduced.)
ƒ If a user interface is high risk, the goal should be to test it early. Same principle as
developing high risk items.
ƒ Some teams have run reliability test early, but for a shorter duration (1st 2 days, then 5, 14
days at the end)

ƒ For the last iteration, most development teams are only be fixing defects and
reacting to customer change requests
 Agenda

ƒ Agile Software Development

ƒ “Traditional” Software Testing

ƒ Everything is different but still the same

ƒ The fight is on

ƒ IBM Agile Project

ƒBringing it all together

27
The Importance of Test Management

ƒ Would you go on a road trip

without a map?

ƒ How would you know where to

go?
ƒ How would you know how long it
would take?
ƒ How would you reroute around
problems?
ƒ How would you know how to do
it better next time?

28
Integrations 29

ƒQuality effects everyone and everything

ƒ Requirements
ƒ Defects
ƒ Project Management
ƒ Lab Management
ƒ Development Environment
ƒ Source Code
Agile Test Plans 30

ƒHave heard this statement from

many..
ƒ We are Agile, we don’t do test plans

ƒWhat really saying

ƒ We don’t create 90 page documents early on in project and hope we
meet later
ƒ Test Plan no a formal deliverable
ƒ Do not create doc before start testing

ƒNot saying
ƒ We have no process/strategy
ƒ Only do Spray and Pray testing
ƒ We have no deliverables
ƒ We have no goals
Agile Test Plans 31

ƒPlan is targeted for product /project

ƒAll effort needs optimized
ƒPriorities based on concepts such as
risk
ƒPlan is not full done before starting
test cycle
ƒIf test Plan is a deliverable, then it is a
generated document/
ƒCreated just in time (grows in detail
as product evolves).
Agile Test Plans 32

ƒControls scope of Test Project

ƒContain test Strategies
ƒCaptures Reports and Metrics to be
delivered
ƒContains coverage objectives
ƒ Requirements
ƒ Platforms
ƒ Schedules
ƒ Risk
ƒMost likely not called a Test Plan
We shape our tools; thereafter, our 33
tools shape us. (McLuhan)
ƒIf you Test you have a plan
ƒ May not admit it

ƒPoint to Information rather than repeat it

ƒ Why best to use a tool or Wiki to capture

ƒCoverage Outline
ƒ Product elements, function, data , schedule, etc

ƒContains coverage objectives

ƒ Requirements
ƒ Platforms
ƒ Schedules
ƒ Risk
 34
Best Practices
ƒ Ensure you are using a sampling of customers and not
designing a product for one specific customer. Design
features that satisfy 80% of the customer use cases.

ƒ Provide constant and frequent feedback and testing.

Include hard data and softer feedback about usability
issues and designs.

ƒ Do not back end load performance testing. Run early

as part of FVT and SVT. Run benchmarks on each
release for comparison.

ƒ It’s imperative that development commits to provide

enough information for each iteration to enable
successful test preparation and collaboration. Test
must commit to help prioritize defects and provide
build identification.
 35

Best Practices
ƒ Avoid using record and playback for GUI automation
for a 1.0 product release due to the amount of user
interface changes. Utilize headless tests or using SDKs
or APIs to drive automation. Test services and
enterprise hardening for your product.

ƒ Use a good test management tool so you can have

multiple versions of your testcases and associate them
with a given iteration or release.

ƒ Use collaboration tools to ensure development and

test teams are constantly in touch and aware of the
latest hot items.

ƒ Define release defining or high level requirements and

track them.
Agile testing 36

ƒSimilarities to today
ƒ Testcases and test plans are living documents and are enhanced
each iteration as features are delivered and coverage changes.
ƒ Limited use of formal entrance and exit criteria and rely on
general inputs required for milestone assessments (weekly builds,
development published themes and goals, customer feedback
etc)
ƒ Focus on all test types as applicable: integration, reliability long
runs, scalability, stress scenarios, functional capability regression,
install/config, system performance, end-to-end customer
scenarios
ƒ Provide input to final quality certification of the software
ƒ Provide “softer” experience report for each iteration as well as
data from defects and execution
Agile testing 37

ƒKey Differences
ƒ Test coverage spread across iterations, with emphasis on
customer scenarios evolving with the addition of features – EARLY
FEEDBACK
ƒ Less “dead” time at the end of the cycle although you may still
have a final SVT phase
ƒ More defects can be addressed in the current release
ƒ SVT is better able to impact new designs and features and is
always working with the product and impacting quality, even
when not doing an actual tracked test pass
Happily Ever After 38

ƒTest Management Necessary in any process

ƒTest Plans critical for controlling test activity
ƒ Degree of formalism may vary
Page
39 39
 Thank you!

For more information:

Brian Massey
Senior Product Manager
IBM/Rational Software
678-248-4523
jbmassey@us.ibm.com
www.ibm.com/rational