SESSION 9 –Other Players in Testing Project • Over the last 20 years, outsource development of one or more key
Needs for Partners
• You might want to get business analysts, help desk or technical
support, target customers or users, or sales and marketing staff
involved in testing to build their confidence in the quality of the
system that’s about to be released— or the testing that your team or
some other team performed on that system. In some cases, the
motivation is political, to make up for credibility deficits in the
development team or your own team of testers. In other cases—
especially customer involvement in acceptance testing— the
motivation is contractual, since many outsource development efforts
include a period of acceptance testing by the customers prior to final
payment.
Partners in Testing Project
Vendors
• When vendors are producing key components for your systems,
clear-headed assessments of component quality and smart testing
strategies are key to mitigating those risks.
• Most vendors take a narrow, targeted view of their testing: they
might test very deeply in certain areas, but they aren’t likely to test
broadly.
• What is the risk to your system’s quality related to your vendors’
components?
– Component irreplaceability.
– Component essentiality.
– Component/system coupling.
– Vendor quality problems.
Testing Service Providers
• Testing service providers include any organization that offers test
services to clients. Most of the time these services come at a fee,
but some organizations provide some specialized services at no
charge to the client. The provider can offer these services on-site
(insourcing), off-site (outsourcing), or both.
• A testing service provider brings several key strengths to the table.
The most important is expertise in test project management and
test engineering.
• Another advantage is that testing service providers can often begin
running tests for you more quickly than you could do it yourself.
• A testing service provider, whether lab or consultancy or whatever,
might also offer expert consulting and training services.
Sales Office
• If you sell a product internationally, you might have a local sales office
or a sales partner (such as a distributor) in various regions.
• As fellow employees, the staff members in a sales office have the
same goals and objectives you do —they have a stake in ensuring that
the test effort is successful.
• Unfortunately, these sales and marketing people might not have
technical sophistication or a particular skill with testing. If you are
responsible for the results of the testing and want specific items
tested, you will need to spell out these details. Any test cases you give
to nontechnical colleagues must be precise and unambiguous.
Users & User-Surrogates
• This category includes business analysts, help desk, customer support, • The outsourced testers must have the ability to adapt their work to
and technical support personnel along with actual target customers
and users.
• Most commonly, these folks participate in alpha, beta, pilot, or
acceptance testing efforts. (One of our clients, though, invites its
customers’ most-expert users of its system to participate in system
test, using their own unique data and workflows).
• As mentioned previously, testing by users and user-surrogates can
result from considerations of credibility, contractual obligation, or
from a need to broaden test coverage.
Outsourcing in Testing Project
How Outsourcing Affects Testing
components in the system has come to dominate software and
hardware systems engineering. The trend started in hardware in the
1990s. RBCS clients like Dell, Hitachi, Hewlett-Packard, and other
computer systems vendors took advantage of cheap yet educated
labor overseas to compete effectively in an increasingly
commoditized market.
• Outsourcing spread slowly into software in the 1990s. The near-
simultaneous bursting of the telecom and dot-com bubbles in 2000
combined with the Y2K and Euro-conversion wind-downs, making
the software industry and its customers more conservative and
price-conscious. By the end of 2002, when after three years into a
spectacular IT downturn computer science enrollments in the
United States fell to less than half of their 1999 levels, price had
become the primary determinant inmost IT project decisions. Mass
outsourcing of software projects took hold, and it continues
unabated to this day.
• As with hardware, commoditization plays a role with software
outsourcing. Software as a Service (SaaS) represents one facet of
commoditized software, and it is clearly a form of pay-as-you-go
outsourcing. The use of open source packages represents another
facet of commoditized software, being a form of pay-nothing-as-
you-go outsourcing (albeit requiring potentially expensive self-
support).
• We also need to consider a from the outside looking in perspective;
i.e., how you manage testing when a constraint is distribution of
some of the testing, all of the testing, and perhaps even the entire
development effort.
Outsourcing Scopes
• Outsource the development, but retain the testing in-house.
• Outsource the development and the testing to one company only.
• Outsource the development and the testing to two different
companies.
• Outsource the development and/or the testing, each to multiple
companies.
Reasons for Outsourcing
• The desire to realize labor and other cost savings.
• The expertise, capital equipment, or geographical advantages of the
outsource organization.
• The need for system or product certification by a qualified service
provider.
• The inability to handle the work in-house, either due to a temporary
spike in workload or a one-off project.
• Organizational or peer pressure on decision-makers to outsource,
whichis the ‘‘everybody’s doing it’’ argument.
• Dissatisfaction with the in-house team’s capability, whether for
service, cost, attitude, quality, or some other reason, which is the ‘‘it
couldn’t be worse, at least it’s cheaper’’ argument.
Success Factors for Outsourcing
• You have to select the right testers for the testing tasks, with the right
skills. This includes test management skills for large chunks of testing
work that will be outsourced.
• The outsourced testers must have access to the right equipment, the
right tools, the right infrastructure, and so forth. If the assumption is
that the outsourced testing will use equipment, tools, and
infrastructure that you have in-house, this will not happen by magic,
but will require careful planning and management of the logistics of
that.
your project and your organization.
• The outsourced testers must have sufficient independence to tell the
straight truth about their results.
Issues for Outsourced Project
• The system being built: This includes not only the transfer of test
releases, release notes, code, and other supporting files, under proper
configuration management control and as part of a solid release
engineering process, but also the change management and project
management elements.
• The test system: Due to reasons of expense or security, it is often
necessary to share test environments with the outsource
 development and, if you have them, outsource testing partners. This
includes the obvious elements of the test environment, but also
access to cohabiting software, affected or linked systems or
databases, and other connected elements that will be present in the
customer or production environment. You’ll need also to be able to
exchange various testware items, such as test data, cases, and test
tools, both to audit the test work of the outsource organizations and
to avoid problems with gaps and overlap in the creation of the
testware. Finally, you also need to coordinate the test processes, at
least the major touchpoints that cross-organizational boundaries.
• Information flows: A tremendous amount of information should flow
across organization boundaries as part of testing in an outsourced
project. This includes project documents, quality risk analysis
documents, estimates and plans, work assignments, bug reports, and,
of course, test results reports. It also includes less-formalized but
equally critical information flows like emails, project discussion
enablers like wikis and newsgroups, and the like.
Risks for Outsourced Project
• Political instability, especially when using outsource organizations in
developing regions
• Time zones, language barriers, and other communication issues.
• Overtime and other off-hours constraints due to lack of access to
the facility, lack of power or other infrastructure during off-hours at
the facility, or physical safety issues associated with being at the
facility during off-hours.
• Infrastructure problems and inadequacies, such as unreliable
Internet connectivity, poor roads and airports, difficulties obtaining
potable water and safe food during site visits, and so forth.
• Skills availability, as mentioned in the earlier section.
• Unforeseen and sometimes abruptly exposed organizational
weakness, including loss of key players due to extreme rates of
turnover, organizational collapse due to governance problems, and
the like.
Conclusion about Outsourcing for Testing
• As discussed in this section, outsourcing does pose a number of
challenges to testing and quality. However, none of those challenges
are fundamentally that much different or harder to manage than the
risks and activities on non-outsourced, collocated projects. The
differences are those of degree more than of kind. So, the diligent test
manager can succeed with outsourced projects if she manages the
overall testing process with the same level of attention to detail that
she would use to manage any other testing effort.
• Software outsourcing matures, testers and test managers are well
positioned to become essential project participants. Outsourcing
might be cheaper than doing projects in-house, but it’s harder, much
more complex, and thus riskier. Since testing, ultimately, is composed
of risk management activities, I suggest you bring your risk
management skills to bear. You might well find that you are the key to
outsourcing success in your organization.
SESSION 10 - Review for Testing Project
Review Principles
• A review is a type of static test. The object being reviewed is not
executed or run during the review. Like any test activity, reviews can
have various objectives.
• Reviews usually precede dynamic tests. They should complement
dynamic tests. Because the cost of a defect increases the longer that
defect remains in the system, reviews should happen as soon as
possible.
• Because reviews are so effective when done properly, organizations
should review all important documents. That includes test
documents: test plans, test cases, quality risk analyses, bug reports,
test status report, you name it.
What Can Happen after a Review?
• There are three possible outcomes.
– The ideal case is that the document is okay as is or
with minor changes.
– Another possibility is that the document requires
some non-trivial changes but not a rereview.
– The most costly outcome—in terms of both effort
and schedule time—is that the document requires
extensive changes and a re-review.
• Now, when that happens, keep in mind that while this is a
costly outcome, it’s less costly than simply ignoring the
serious problems and then dealing with them during
component, integration, system, or—worse yet—acceptance
testing.
Essential Roles and Responsibilities
• During a formal review, there are some essential roles and
responsibilities:
• The manager. The manager allocates resources, schedules reviews,
and the like. However, the manager might not be allowed to attend
based on the review type.
• The moderator or leader: This is the chair of the review meeting.
• The author: This is the person who wrote the item under review. A
review meeting, done properly, should not be a sad or humiliating
experience for the author.
• The reviewers: These are the people who examine the item under
review, possibly finding defects in it. Reviewers can play specialized
roles based on their expertise or based on some type of defect they
should target.
• The scribe or secretary or recorder: This is the person who writes
down the findings.
Types of Review
• At the lowest level of formality (and, usually, defect removal
effectiveness), we find the informal review. This can be as simple as
two people, the author and a colleague, discussing a design
document over the phone.
• Technical reviews are more formalized, but still not highly formal.
• Walk-throughs are reviews where the author is the moderator and
the item under review itself is the agenda. That is, the author leads
the review, and in the review, the reviewers go section by section
through the item under review.
• Inspections are the most formalized reviews. The roles are well
defined. Managers may not attend. The author may be neither
moderator nor secretary. A reader is typically involved.
Six Phases for Formal Review
Preparing the Review
• You don’t have to wait until a document is done before you start
reviewing it. You can and should review early drafts or partial
documents when you’re dealing with something critical. This can help
to identify and prevent patterns of defects before they are built into
the whole document.
• That said, make sure you have some rules about what it means for
something to be ready for review.
• Checklists are helpful for reviews. It’s too easy to forget important
areas without them. Have some checklists.
Marick’s Code Review Checklist
• Technical test analysts are likely to be invited to code reviews. So, let’s
go through Brian Marick’s code review checklist, which he calls a
“question catalog.” This catalog has several categories of questions
that developers should ask themselves when going through their
code. These questions are useful for many procedural and object-
oriented programming languages, though in some cases certain
questions might not apply.
• However, customization based on your own experience, and your
organization’s needs, is encouraged.
• For variable declarations, Marick says we should ask the following
questions:
• Are the literal values correct? How do we know?
 • Has every single variable been set to a known value
before first use? When the code changes, it is easy to
miss changing these.
• Have we picked the right data type for the need? Can
the value ever go negative?
• For each data item and operations on data items, Marick says
we should ask the following questions:
• Are all strings NULL terminated? If we have shortened or
lengthened a string, or processed it in any way, did the
final byte get changed?
• Did we check every assignment to a buffer for length?
• When using bitfields, are our manipulations (shifts,
rotates, etc.) going to be portable to other architectures
and endian schemes?
• Does every sizeof() function call actually go to the object
we meant it to?
• For every allocation, deallocation, and reallocation of
memory, Marick says we should ask the following questions:
• Is the amount of memory sufficient to the purpose
without being wasteful?
• How will the memory be initialized?
• Are all fields being initialized correctly if it is a complex
data structure?
• Is the memory freed correctly after use?
• Do we ever have side effects from static storage in
functions or methods?
• After reallocating memory, do we still have any pointers to
the old memory
• location?
• Is there any chance that the memory might be freed
multiple times?
• After deallocation, are there still pointers to the memory?
• Are we mistakenly freeing data we don’t mean to?
• Is it possible that the pointer we are using to free the
memory is already NULL?
• For all operations on files, Marick says we should ask the
following questions:
• Do we have a way of ensuring that each temp file we create
is unique?
• Is it possible to reuse a file pointer while it is pointing to an
open file?
• Do we recover each file handle when we are done with it?
• Do we close each file explicitly when we are done with it?
• For every computation, Marick says we should ask the
following questions:
- Are parentheses correct? Do they mean what we want them
to mean?
- When using synchronization, are we updating variables in the
critical sections together?
- Do we allow division by zero to occur?
- Are floating point numbers compared for exact equality?
• For every operation that involves a pointer, Marick says we
should ask the following questions:
• Is there any place in the code where we might try to
dereference a NULL pointer?
• When dealing with objects, do we want to copy
pointers (shallow copy) or content (deep copy)?
• For all assignments, Marick says we should ask the following
question:
• Are we assigning dissimilar data types where we can
lose precision?
• For every function call, Marick says we should ask the
following questions:
• Was the correct function with the correct arguments
called?
• Are the preconditions of the function actually met?
• Finally, Marick provides a couple of miscellaneous questions:
• Have we removed all of the debug code and bogus error
messages?
• Does the program have a specific return value when exiting?
OpenLaszlo Code Review Checklist
• In the previous section, we discussed Marick’s questions that should
be asked about the code itself. In this section, we will discuss
questions that should be asked about the changes to the system.
These are essentially meta-questions about the changes that
occurred during maintenance. These come from the OpenLaszlo
website.
• For all changes in code, here are the main questions we should ask:
• Do we understand all of the code changes that were
made and the reasons for them?
• Are there test cases for all changes? Have they been
run?
• Were the changes formally documented as per our
guidelines?
• Were any unauthorized changes slipped in?
• In terms of coding standards, here are some additional questions to
ask (assumingyou are not enforcing coding standards via static
analysis):
• Do all of the code changes meet our standards and guidelines?
If not, why not?
• Are all data values to be passed parameterized correctly? In
terms of design changes, here are the questions to ask:
• Do you understand the design changes and the reasons they
were made?
• Does the actual implementation match the designs?
• Here are the maintainability questions to ask:
• Are there enough comments? Are they correct and sufficient?
• Are all variables documented with enough information to
understand why they were chosen?
• Finally, here are the documentation questions included in the
OpenLaszlo checklist:
• Are all command-line arguments documented?
• Are all environmental variables needed to run the system
defined and documented?
• Has all user-facing functionality been documented in the user
manual and help file?
• Does the implementation match the documentation?
Developing a Review Program
• Reviews are an effective tool used along with execution-based testing
to support defect detection, increased software quality, and customer
satisfaction. Reviews should be used for evaluating newly developing
products as well as new releases or versions of existing software. If
reviews are conducted on the software artifacts as they are
developed throughout the software life cycle, they serve as filters for
each artifact.
• A multiple set of filters allows an organization to separate out and
eliminate defects, inconsistencies, ambiguities, and low-quality
components early in the software life cycle. If we compare the
process of defect detection during reviews and execution-based
testing/debugging we can see that the review process may be more
efficient for detecting, locating, and repairing these defects, especially
in the code, for the following reasons:
1. When testing software, unexpected behavior is observed
because of a defect(s) in the code. The symptomatic
information is what the developer works with to find and repair
(fix) the defect.
2. Reviews also have the advantage of a two-pass approach for
defect detection. Pass 1 has individuals first reading the
reviewed item and pass 2 has the item read by the group as a
whole. If one individual reviewer did not identify a defect or a
problem, others in the group are likely to find it.
3. Inspectors have the advantage of the checklist which calls their
attention to specific areas that are defect prone. These are
important clues. Testers/developers may not have such
information available.
Review-related Policies
(i) testing policies with an emphasis on defect detection and
quality, and measurements for controlling and monitoring;
(ii) a test organization with staff devoted to defect detection and
quality issues;
 (iii) policies and standards that define requirements, design, test
plan, and other documents;
(iv) organizational culture with a focus on quality products and
quality processes.
• Review policies should specify when reviews should take
place, what is to be reviewed, types of reviews that will take
place, who is responsible, what training is required and what
the review deliverables are.
• Review procedures should define the steps and phases for
each type of review.
• Policies should ensure that each project has an associated
project plan, test plan, configuration management plan, and a
review plan, and/or a software quality assurance plan.
• Project plans and the review plans should ensure that
adequate time and resources are available for reviews and
that cycle time is set aside for reviews.
• Managers need to follow-up and enforce the stated policies.
Only strong managerial commitment will lead to a successful
review program.
Review Plan Components
• Reviews are development and maintenance activities that
require time and resources. They should be planned so that
there is a place for them in the project schedule. An
organization should develop a review plan template that can
be applied to all software projects. The template should
specify the following items for inclusion in the review plan.
– review goals;
– items being reviewed;
– preconditions for the review;
– roles, team size, participants;
– training requirements;
– review steps;
– checklists and other related documents to be disturbed to
participants;
– time requirements;
– the nature of the review log and summary report;
– rework and follow-up.
Review Goals
• As in the test plan or any other type of plan, the review
planner should specify the goals to be accomplished by the
review and include:
i. identification of problem components or
components in the software artifact that need
improvement,
ii. identification of specific errors or defects in the
software artifact,
iii. ensuring that the artifact conforms to organizational
standards, and
iv. communication to the staff about the nature of the
product being developed.
• Additional goals might be to establish traceability with other
project documents, and familiarization with the item being
reviewed. Goals for inspections and walkthroughs are usually
different; those of walkthroughs are more limited in scope
and are usually confined to identification of defects.
Preconditions and Items to Be Reviewed
• Given the principal goals of a technical review—early defect
detection, identification of problem areas, and familiarization
with software artifacts—many software items are candidates
for review. In many organizations the items selected for
review include:
– requirements documents;
– design documents;
– code;
– test plans (for the multiple levels);
– user manuals;
– training manuals;
– standards documents.
• The preconditions need to be described in the review policy
statement and specified in the review plan for an item.
General preconditions for a review are:
– the review of an item(s) is a required activity in the project
plan. (Unplanned reviews are also possible at the request
of management, SQA or software engineers. Review policy
statements should include the conditions for holding an
unplanned review.)
– a statement of objectives for the review has been
developed;
– the individuals responsible for developing the reviewed
item indicate readiness for the review;
– the review leader believes that the item to be reviewed is
sufficiently complete for the review to be useful.
• The review planner must also keep in mind that a given item
to be reviewed may be too large and complex for a single
review meeting. The smart planner partitions the review item
into components that are of a size and complexity that allows
them to be reviewed in 1–2 hours. This is the time range in
which most reviewers have maximum effectiveness. For
example, the design document for a procedure-oriented
system may be reviewed in parts that encompass:
– the overall architectural design;
– data items and module interface design;
– component design.
• If the architectural design is complex and/or the number of
components is large, then multiple design review sessions
should be scheduled for each. The project plan should have
time allocated for this.
Roles, Participants, Team Size, and Time Requirements
• Two major roles that need filling for a successful review are (i)
a leader or moderator, and (ii) a recorder.
• Some of the responsibilities of the moderator have been
described. These include planning the reviews, managing the
review meeting, and issuing the review report. Because of
these responsibilities the moderator plays an important role;
the success of the review depends on the experience and
expertise of the moderator.
• Reviewing a software item is a tedious process and requires
great attention to details. The moderator needs to be sure
that all are prepared for the review and that the review
meeting stays on track. Reviewers often tire and become less
effective at detecting errors if the review time period is too
long and the item is too complex for a single review meeting.
The moderator/planner must ensure that a time period is
selected that is appropriate for the size and complexity of the
item under review.
• There is no set value for a review time period, but a rule of
thumb advises that a review session should not be longer than
2 hours. Review sessions can be scheduled over 2-hour time
periods separated by breaks. The time allocated for a review
should be adequate enough to ensure that the material under
review can be adequately covered.
• The review recorder has the responsibility for documenting
defects, and recording review findings and recommendations,
Other roles may include a reader who reads or presents the
item under review. Readers are usually the authors or
preparers of the item under review. The author(s) is
responsible for performing any rework on the reviewed item.
In a walkthrough type of review, the author may serve as the
moderator, but this is not true for an inspection. All reviewers
should be trained in the review process.
• The size of the review team will vary depending type, size, and
complexity of the item under review. Again, as with time,
there is no fixed size for a review team. In most cases a size
between 3 and 7 is a rule of thumb, but that depends on the
items under review and the experience level of the review
team. Of special importance is the experience of the review
moderator who is responsible for ensuring the material is
covered, the review meeting stays on track, and review
 outputs are produced. The minimal team size of 3 ensures
that the review will be public.
Review Procedures
• For each type of review that an organization wishes to implement,
there should be a set of standardized steps that define the given
review procedure. These are initiation, preparation, inspection
meeting, reporting results, and rework and follow-up. For each step in
the procedure the activities and tasks for all the reviewer participants
should be defined. The review plan should refer to the standardized
procedures where applicable.
Review Training
• Review participants need training to be effective. Responsibility for
reviewer training classes usually belongs to the internal technical
training staff. Alternatively, an organization may decide to send its
review trainees to external training courses run by commercial
institutions. Review participants, and especially those who will be
review leaders, need the training. Test specialists should also receive
review training.
Review Checklists
• Checklists are very important for inspectors. They provide structure
and an agenda for the review meeting. They guide the review
activities, identify focus areas for discussion and evaluation, ensure all
relevant items are covered, and help to frame review record keeping
and measurement. Reviews are really a two-step process: (i) reviews
by individuals, and (ii) reviews by the group. The checklist plays its
important role in both steps. The first step involves the individual
reviewer and the review material. Prior to the review meeting each
individual must be provided with the materials to review and the
checklist of items. It is his responsibility to do his homework and
individually inspect that document using the checklist as a guide, and
to document any problems he encounters. When they attend the
group meeting which is the second review step, each reviewer should
bring his or her individual list of defect/problems, and as each item on
the checklist is discussed they should comment. Finally, the reviewers
need to come to a consensus on what needs to be fixed and what
remains unchanged.
Example Components for an Inspection Checklist
• The first column lists all the defect types or potential problem areas
that may occur in the item under review. Sources for these defect
types are usually data from past projects. Abbreviations for detect/
problem types can be developed to simplify the checklist forms.
Status refers to coverage during the review meeting—has the item
been discussed? If so, a check mark is placed in the column.
• Major or minor are the two severity or impact levels shown here. Each
organization needs to decide on the severity levels that work for
them. Using this simple severity scale, a defect or problem that is
classified as major has a large impact on product quality; it can cause
failure or deviation from specification. A minor problem has a small
impact on these; in general, it would affect a nonfunctional aspect of
the software. The letters M, I, and S indicate whether a checklist item
is missing (M), incorrect (I), or superfluous (S).
Extended Checklist
• In addition to covering the items on the general document checklist as
shown in previous table, the following items should be included in the
checklist for a requirements review.
– completeness (have all functional and quality requirements
described in the problem statement been included?);
– correctness (do the requirements reflect the user’s needs? are
they stated without error?);
– consistency (do any requirements contradict each other?);
– clarity (it is very important to identify and clarify any ambiguous
requirements);
– relevance (is the requirement pertinent to the problem area?
requirements should not be superfluous);
– redundancy (a requirement may be repeated; if it is a duplicate it
should be combined with an equivalent one);
– testability (can each requirement be covered successfully with
one or more test cases? can tests determine if the requirement
has been satisfied?);
– feasibility (are requirements implementable given the conditions
under which the project will progress?).
Design Review
• Some specific items that should be checked for at a design review
– a description of the design technique used;
– an explanation of the design notation used;
– evaluation of design alternatives;
– quality of the high-level architectural model;
– description of module interfaces;
– quality of the user interface;
 – quality of the user help facilities;
– identification of execution criteria and operational sequences;
– clear description of interfaces between this system and other
software and hardware systems;
– coverage of all functional requirements by design elements;
– coverage of all quality requirements, for example, ease of use,
portability, maintainability, security, readability, adaptability,
performance requirements (storage, response time) by design
elements;
– reusability of design components;
– Testability.
Detailed Design Review
• For reviewing detailed design the following focus areas should also be
revisited:
– encapsulation, information hiding and inheritance;
– module cohesion and coupling;
– quality of module interface description;
– module reuse.

Review Reports
• The review reports should contain the following information.
– For inspections—the group checklist with all items covered and
comments relating to each item.
– For inspections—a status, or summary, report (described below)
signed by all participants.
– A list of defects found, and classified by type and frequency. Each
defect should be cross-referenced to the line, pages, or figure in the
reviewed document where it occurs.
– Review metric data.
IEEE Standards Inspection Report
• IEEE standards suggest that the inspection report contain vital
data such as [8]:
– number of participants in the review;
– the duration of the meeting;
– size of the item being reviewed (usually LOC or number of
pages);
– total preparation time for the inspection team;
– status of the reviewed item;
– estimate of rework effort and the estimated date for
completion of the rework.
Purpose of Review
• A final important item to note: The purpose of a review is to evaluate
a software artifact, not the developer or author of the artifact.
Reviews should not be used to evaluate the perforance of a software
Test Plan Checklist
analyst, developer, designer, or tester.This important point should be
well established in the review policy. It is essential to adhere to this
policy for the review process to work. If authors of software artifacts
believe they are being evaluated as individuals, the objective and
impartial nature of the review will change, and its effectiveness in
revealing problems will be minimized

SESSION 11 - Test Lab

Do You Need a Test Lab?
• Not every test organization needs a test lab. Some organizations
need a lab only at certain times. Others can’t test without a test lab.
Because setting up and especially maintaining a decent test lab is an
expensive proposition, you should carefully evaluate whether or not
you actually need a lab.
Considering a Test Lab
• Do you need large test tools such as heat chambers? Are some of
your test tools non-portable ones—a shock and vibration table, for
example—that need a special permanent location?
• Is a special environment required? If your test platforms have strict
environmental requirements (as servers do) or unusual voltage
needs (as telephone switches do), you will need at least a server
room, if not a separate test lab.
• Is security an issue? For example, when testing new software or
hardware, confidentiality is a big concern. Secrets stay secret
onlywhen as few people as possible know about them. Moreover, if
you perform compatibility testing with a variety of software
packages, the CDs and DVDs with the software are valuable, as are
the CPUs, laptop computers, tape drives, hard drives, and the like.
Keeping them in a restricted access test lab— especially in a locked
cabinet— can save thousands of dollars in shrinkage (from loss,
damage, or theft) over the years.
• Do you need to prevent non-testers from fiddling with your test
environment? You might find that some individuals from other
teams can’t help themselves; they insist on loading quick patches or
trying to hook up some device to the test platform ‘‘just to see if it
works.’’ They then forget to undo whatever they did. If you work
with people like this, the formalism of a test lab might deter such
well-intentioned but counterproductive hacking on your test
platforms.
• Do you need access to the test facility for an extended period of
time? Can multiple projects— concurrent, sequential, or both—
leverage the cost of the lab to lower the total cost to the
organization? Better yet, can you make the test lab a profit center
by selling tools and services from the lab to outside, noncompeting
companies? (This is less unlikely than you might expect. For
example, if you have a printer compatibility test lab for
SpeedyWriter, you might sell printer compatibility test services to
other companies.)
• If you really need a test lab, you will be able to make a business case
for it based on previous (and possibly other) factors. Try preparing a
budget specifically for the test lab; you’ll be surprised at how much
it can cost. Remember, too, that money and effort are required not
only to set up a lab, but also to maintain it. If the costs seem
prohibitive, you almost always have alternatives. By carefully using
external resources such as third-party test labs and your vendors,
you can leverage their investments in labs and lab materiel to
minimize or eliminate the need for your own.
Selecting and Planning a Lab Area
• Once you’ve decided that a test lab is necessary, you’ll need to select
a location for the lab and plan its configuration. As you consider the
factors outlined here, try to sketch a scaled floor plan of the lab, along
the lines of the example shown in next Figure. Make this floor plan as
large as possible if you sketch it on paper, or use graphing or drawing
software such as Visio. Either way, you should count on developing
the floor plan iteratively; you might even need to start all over if you
discover that the space you initially selected is unsuitable.
Size. Is the potential lab space large enough? When the shape of the
room is anything other than square or rectangular and wide, you must
consider size in conjunction with layout — that is, you must look at the
actual working space. Also consider the height of the ceiling. How high
will you want your racks of equipment? Also consider the wiring ladders
or wiring racks that you might need. Pay attention to the doors—
everything in the lab must come in and, eventually, go out through
them. Begin your sketch of the lab by drawing the lab area, including
the location of the doors and the direction in which they open.
Lighting. Windows can provide both aesthetic benefits and a welcome
break for testers. You’ll want to have tinted windows in your lab with
effective shades installed. Also note the level of the windows: if they are
at ground level or close to it, they can create a security problem. As for
artificial lighting, fluorescent lights are the most common type and are
probably the best. Incandescent lights tend to throw light in limited
areas. Spot lights, track lights, and the like look cool, but they are a
better choice for conference rooms than for labs, again because of the
type of light they give off. Even and consistent lighting is important
when testers are reading documents and screens or evaluating the
quality of a display. On your floor plan, indicate the locations of both
lighting fixtures and windows (if any).
Layout. When you select a lab space, keep in mind the types of tables
and equipment racks you need to install. If you rack-mount most of your
equipment, or if it is freestanding, a square room might work very well.
If you use tables or desks, a rectangular room might be a better choice,
as long as it is wide enough. Remember to leave enough open space to
allow people to come and go freely. Populate your floor plan sketch
with a scale layout of the chairs, desks, cabinets, shelves, tables, racks,
and equipment you intend to install. To keep your floor plan less
cluttered, you can use a legend to describe some of these objects.
Climate control. The test lab must have sufficient air conditioning and
heating to preserve a stable, normal operating temperature and
humidity for your equipment and your testers. Although the testers
might go home, the equipment does not — which means continuous
(24-hour, year-round) climate control. Your lab might also need
automated shutdown capabilities if the temperature or humidity
exceeds operational limits. If you have multiple labs or workspaces on
the same system, ensure that each lab has its own thermostat. Locate
the thermostat on your floor plan.
Fire safety and prevention. Every workplace should have a smoke or fire
detector, either battery-powered or backed up by a battery. You should
test this detector regularly. To control a fire, a test lab must have, ready
at hand, portable fire extinguishers that are electrically safe and rapidly
effective. If the lab contains large equipment such as mainframes or
telephony switches, you will also need sophisticated, automatic fire
suppression equipment that can put out a large electrical fire on its own
while staff members attend to their personal safety. Include the fire
detection and extinguishing equipment in your sketch, and make sure
that your layout doesn’t impede access to these devices or impede their
use or flow. If objects in the lab get in the way, you should revisit your
layout.
Power. You’ll need appropriate electrical power in one form or another
(120 VAC, 240 VAC, 480 VAC, 48 VDC, and so on) in any test lab, and
you’ll need the power outlets or connections in the right places. In
addition, the incoming power must be conditioned and uninterruptible,
immune from the spikes, surges, sags, brownouts, and blackouts that
can plague the power grid. If it isn’t, you will need to provide such
conditioning and backup power in the lab, or be prepared to suffer the
consequences. Indicate all the various power outlets on your floor plan.
Static. If you decide to carpet your lab, you will need to take extra
precautions to inhibit static. Tile, linoleum, cement, and raised floors
tend not to suffer from this problem to the same degree, but even
clothing can generate static. The lab should contain static mats and
other grounded metal objects to help testers dissipate static when they
accumulate it.
Facilities. Facilities for your staff, such as restrooms, stairways,
elevators, and so forth (including handicap accessibility) are important
not only because you might be legally obligated to provide them, but
also because they affect the productivity of the team. In addition,
remember that you will need to connect the lab to the outside world
using telephone lines (PSTN, BRI ISDN, PRI ISDN, T1, OC3, and so on);
network infrastructure such as category-5 Ethernet, category-6
Ethernet, optical, WWAN, or ATM; and possibly other connections. If
the proposed lab site does not already have the appropriate wiring
installed, you will need to build it in, possibly at considerable expense.
(A raised floor configuration can make wiring within the lab painless, but
the wiring must still be brought into the lab space, either through a wall
or through a crawl space above the ceiling or below the floor.) You
should also identify connections for running water if your planned tests
require water for cooling. Indicate all facilities and connections on your
drawing.
Test Lab Inventory
• Software: Operating system, applications, test tools and utilities
• Hardware: PC cards, monitor/video cards, printer/scanners,
connectivity cards, data storage, surge protectors/UPS units,
reference platforms, cables, networking
• Consumables: Computer media, desk necessities, printer supplies,
• Furnishing: Copiers and printers, shredder, benches, desks, stools,
chairs, mouse pads, static pads
• Tools: Specialized hardware, test management software, computer
toolkit,
• Reference Materials: Best practices books, industry or government
standards
SESSION 12 - Test Management – Closing
Criteria for Test Completion
1. All the Planned Tests That Were Developed Have Been
Executed and Passed.
– This may be the weakest criterion. It does not take into
account the actual dynamics of the testing effort, for example,
the types of defects found and their level of severity. Clues
from analysis of the test cases and defects found may indicate
that there are more defects in the code that the planned test
cases have not uncovered. These may be ignored by the
testers if this stop-test criteria is used in isolation.
2. All Specified Coverage Goals Have Been Met.
– An organization can stop testing when it meets its coverage
goals as specified in the test plan. For example, using white
box coverage goals we can say that we have completed unit
test when we have reached 100% branch coverage for all
units. Using another coverage category, we can say we have
completed system testing when all the requirements have
been covered by our tests. The graphs prepared for the
weekly status meetings can be applied here to show progress 3. Incident description: this should describe time and date, testers,
and to extrapolate to a completion date. The graphs will show
the growth of degree of coverage over the time.
3. The Detection of a Specific Number of Defects Has Been
Accomplished.
– This approach requires defect data from past releases or
similar projects. The defect distribution and total defects is
known for these projects, and is applied to make estimates of
the number and types of defects for the current project. Using IEEE Standard for Test Summary Report
this type of data is very risky, since it assumes the current
software will be built, tested, and behave like the past
projects. This is not always true. Many projects and their
development environments are not as similar as believed, and
making this assumption could be disastrous. Therefore, using 3. Comprehensiveness assessment: the document author discusses the
this stop-criterion on its own carries high risks.
4. The Rates of Defect Detection for a Certain Time Period Have
Fallen Below a Specified Level.
– The manager can use graphs that plot the number of defects
detected per unit time. A graph such as Figure 9.5, augmented
with the severity level of the defects found, is useful. When
the rate of detection of defects of a severity rating under
some specified threshold value falls below that rate threshold,
testing can be stopped. For example, a stop-test criterion
could be stated as: “We stop testing when we find 5 defects
or less, with impact equal to, or below severity level 3, per
week.” Selecting a defect detection rate threshold can be
based on data from past projects.
5. Fault Seeding Ratios Are Favorable.
– The technique is based on intentionally inserting a known set
of defects into a program. This provides support for a stop-
test decision. It is assumed that the inserted set of defects are
typical defects; that is, they are of the same type, occur at the
same frequency, and have the same impact as the actual
defects in the code. One way of selecting such a set of defects
is to use historical defect data from past releases or similar
projects.
Documentation – Test Log
IEEE Standard for Test Log
• Test Log Identifier
– Each test log should have a unique identifier.
• Description
– In the description section the tester should identify the
items being tested, their version/revision number, and
their associated Test Item/Transmittal Report. The
environment in which the test is conducted should be
described including hardware and operating system
details.
• Activity and Event Entries
– The tester should provide dates and names of test log
authors for each event and activity. This section should also
contain:
• Execution description: Provide a test procedure identifier
and alsothe names and functions of personnel involved in
the test.
• Procedure results: For each execution, record the results
and the location of the output. Also report pass/fail status.
• Environmental information: Provide any environmental
conditions specific to this test.
• Anomalous events: Any events occurring before/after an
unexpected event should be recorded. If a tester is unable
to start or compete a test procedure, details relating to
these happenings should be recorded (e.g., a power
failure or operating system crash).
• Incident report identifiers: Record the identifiers of
incident reports generated while the test is being
executed.
IEEE Standard for Test Incident Report
1. Test Incident Report identifier: to uniquely identify this report.
2. Summary: to identify the test items involved, the test procedures,
test cases, and test log associated with this report.
observers, environment, inputs, expected outputs, actual outputs,
anomalies, procedure step, environment, and attempts to repeat.
Any other information useful for the developers who will repair the
code should be included.
4. Impact: what impact will this incident have on the testing effort, the
test plans, the test procedures, and the test cases? A severity rating
should be inserted here.
1. Test Summary Report identifier: to uniquely identify this report.
2. Variances: these are descriptions of any variances of the test items
from their original design. Deviations and reasons for the deviation
from the test plan, test procedures, and test designs are discussed.
comprehensiveness of the test effort as compared to test objectives
 and test completeness criteria as described in the test plan. Any
features or combination of features that were not as fully tested as
was planned should be identified.
4. Summary of results: the document author summarizes the testing
results. All resolved incidents and their solutions should be
described. Unresolved incidents should be recorded.
5. Evaluation: in this section the author evaluates each test item based
on test results. Did it pass/fail the tests? If it failed, what was the
level of severity of the failure?
6. Summary of activities: all testing activities and events are
summarized. Resource consumption, actual task durations, and
hardware and software tool usage should be recorded.
7. Approvals: the names of all persons who are needed to approve this
document are listed with space for signatures and dates.
Testing without Documentation
• In order to design, develop, and run tests, you need what’s often
referred to as a test oracle, something that tells you what the
expected, correct result of a specific test should be. Specifications,
requirements, business rules, marketing road maps, and other such
documents frequently play this role. However, what if you receive
no formal information that explains what the system under test
should do?
• You should carefully consider whether your company is ready for
formal processes before you insist on requirements or design
specifications and other accoutrements of mature development
projects.
• This situation is exacerbated by the increasing popularity of Agile
methodologies. 5 Some of the people behind Agile approaches have
endorsed a set of principles that include the following, ‘‘The most
efficient and effective method of conveying information to and
within a development team is face-to-face conversation.’’ As you
can imagine— if you are not already dealing with this situation—
such a principle discounts the role of written specifications,
encouraging instead an on-going dialog about the correct behavior
of a system. In other words, the outcomes of discussions between
stakeholders represent correct behavior.
• One thing to keep in mind about this situation is that you are
definitely not alone. Many people are struggling with the right
amount of documentation to gather, and errors are made on both
sides.
Options for Testing without Specifications
• If you are testing a commercial product, remember that you
have the benefit of competitors. Because your customers will
expect your product to behave substantially like the products
of your competitors, these competitive products are, in a
sense, your test oracle. In compatibility test labs, for example,
most projects have a reference platform — a competitor’s
system, against which the system under test is being
positioned, in the hope of demolishing it in the marketplace.
• If your technical colleagues won’t tell you what the product
should do, perhaps your friends in sales and marketing will. In
my experience, sales and marketing people live to create
glitzy presentations showing where the product line is going.
Although they can be general and imprecise, these documents
might tell you which features and capabilities the product
should support. If you’re testing a product for which questions
about supported features are harder to answer than
questions regarding correct behavior, these documents might
suffice for a somewhat vague but useful oracle.
• Ask your customer-support colleagues. Your colleagues in
customer support might not have much information about
what the product should do, but they probably know what
they don’t want the product to do. Since your testing stands
between them and the hellish scenario outlined in the
previous section, they are usually happy to tell you.
• Unless the product is truly unique, you can use inductive
reasoning to figure out what constitutes reasonable
expectations and correct behavior in many cases. The generic
categories into which products fit tell you a lot about what the
products are supposed to do: a word processor, a Web
browser, a PC, a laptop, a server, an operating system. Some
esoteric questions might arise, but a core dump, a system
crash, a burning CPU, garbage on the screen, an error
message in the wrong language, and abysmal performance
are indisputably bugs.
• If in doubt, you should consider any suspect behavior buggy.
Because you don’t have a crisp way of determining pass and
fail conditions, you will make mistakes in result interpretation.
Remember that calling correct behavior a bug and working
through the bug life cycle is less detrimental to product
quality than failing to report questionable behavior that does
turn out to be a bug. Be sure to file bug reports when
questions arise.
Layoffs and Liquidation
• Historically, a large number of testing and quality assurance
organizations are disbanded within two years of their
formation. Informal discussions on Internet testing discussion
groups during the early-2000s recession suggested that
testers bear a disproportionate burden in layoffs, especially in
economic downturns.
• For obvious reasons, companies don’t tend to post big notices
around the cube farm six months before a layoff, saying, ‘‘Get
your resumes ready; here comes Chainsaw Al.’’ In some
instances, even the line managers might not know what’s
coming, although they usually do get a few clues.
Worrisome Warning Signs of Layoffs and Liquidation
• Being asked to participate in an employee ranking exercise, especially
if this procedure involves every manager in the company. I have
never seen these rankings used for anything other than whacking
the bottom rungs off the ladder.
• Noticing a decline in your company’s revenues. In one firm, everyone
knew that layoffs were coming when their biggest client stopped
signing up new work and started to withdraw existing work. In
another company, people were laid off from departments that still
produced revenue, while the people who worked on a struggling,
unprofitable product survived. Why? The company saw the success
of that product as a ‘‘bet the farm’’ proposition; some bystanders
lost that bet.
• Noticing a sudden, unexplainable invasion of accountants or
consultants, especially if they’re working with the human resources
department. Ask yourself whether the task at hand is figuring out
severance packages.
• Seeing the human resources department working long hours, even
though no hiring or other obviously HR-related events are ongoing.
As with the item above, there’s a possibility that the long hours
involve organizing a layoff.
• Hearing any rumors of a layoff list. It might well include you or
members of your test team. A test manager I know heard from
other managers that they had seen a list of developers slated to go.
He naıvely assumed that this couldn’t mean testers, without asking
himself how the other managers would choose to drop him a hint if
it did mean testers.
• Seeing signs that the independent test team will be reorganized into
smaller units that will be integrated into the individual development
teams. At least one or two positions are likely to be overhead should
that happen, especially the test manager’s position.
Information for Testing Team
• the workload carried out thus far and what is remaining;
• quality level of services carried out;
• coverage of requirements, of specifications, and of test
conditions (depending on the project progress, we will speak
about test case design or test case execution);
• the number of identified, fixed, or to-be-retested defects and
their impact in terms of regression testing;
• components with more defects than others, in order to adjust
testing;
• the number of executed test cases, those successfully carried
out, the blocked ones, those at fault, and associated
percentages;
• identified defects, their impacts and impacted modules;
• planned test delivery dates; which this happens, and the dashboard is the format in which
• quality level of delivered software (number and types of the information is transmitted.
defects) per version. • Therefore, to succeed with a dashboard, we need to know
Information for Testing Management who the key stakeholders are, what questions they need us to
• planned software delivery dates by the development team for help them answer, and how best to communicate those
testing, in order to anticipate required test environment answers to them. You already know who the key stakeholders
resets and smoke tests; are: the people involved in your quality risk analysis, the
• functionalities, specifications, or requirement changes people who reviewed and commented on your test plan, the
compared to the initial requirements, in order to anticipate peer and upper managers discussed in this chapter, and the
their impact on tests, whether they are new tests to be people who will attend the project status meetings.
designed or existing tests to be modified or removed; • Once you have a first draft of the dashboard, take it back to your
• number of defects detected per period of time and number of stakeholders. Explain the charts and what they mean. Make sure
duplicated or rejected defects, and by failure level the that you also explain the limitations of the charts.
average correction time; • As you use the charts in status reporting, make sure that you
• effort spent compared to the effort planned and the reasons continue to manage and fine-tune the dashboard. Ask your
for any differences, to identify whether are attributable to the stakeholders about the dashboard. Is this providing useful
test team, the development team, or to other causes; information to you? Are you getting it often enough? Assess the
• average test design, test implementation, and test execution credibility and accuracy of your status reports, too. Do you feel
effort, as well as the workload and average cost of the defects that people are taking your findings seriously? Are they acting on
Information for Higher Level Hierarchy the reports? Do the reports paint a truthful picture of project
• the identified quality level; status based on accurate data?
• use of the planned effort and any difference between the Reports and Their Target Audience
planned and actual effort;
• planned delivery dates;
• improvement of the application maturity;
• evolution of the costs and efforts (planned, carried out or
remained to be done);
• evaluation of the effectiveness and maturity of the processes,
as well as process improvement actions that need to be
considered.
• system testing dates, in order to be present if needed;
• provisional date for the beginning of acceptance testing;
• planned delivery date to production or to the market;
• requirements or functionalities postponed to a later delivery
date or subject to limitation (e.g., of performances).
Presenting Bad Results
• In ancient times, the messenger who brought bad news was
sometimes executed, suggesting a human tendency that
remains to this day. When you come to a meeting or
approach a developer with news of bugs or unworkable test
equipment, the first response might be defensiveness, anger,
denial, or attack.
• As dysfunctional as these behaviors are, you will have to deal
with them. Even if others recognize the attacks as more
appropriately directed at the problem rather than at the
reporter (you), they probably won’t leap to your defense.
After all, they have their own problems, and getting involved
in your quarrels will antagonize people. While you can’t make
the attacks go away, you can take certain courses of action to
make the situation better—or worse. SESSION 13 - Implementation
• Likewise, you should guard against melodramatic reporting of System Implementation
results. It’s important to maintain a sense of perspective • At some point, testing is complete and the project team and project
about how a bug will actually affect a customer or user, which manager then become responsible for ensuring that the product or
can differ significantly from how it affects you, your testers, system is transferred successfully from the development and test
and your test system. environment to the operational environment of the customer or
• Although a bug can appear quite dangerous to testers, organization.
developers often see the same bug as benign. While the truth • This transfer requires a tactical approach that is defined during the
lies somewhere between the extremes, you can easily come planning stages of the project, but the actual implementation
off as an alarmist. Worse yet, you can be accused of not being activities can be a stressful time for all the stakeholders involved.
a team player if you perpetually hold the product up to Choosing an inappropriate implementation approach can negatively
unrealistically high standards of quality and rail against impact the project’s remaining schedule and budget.
moving the schedule forward because the product isn’t ready • In general, the project team can take one of three implementation
yet. approaches: (a) direct cutover, (b) parallel, and (c) phased.
Using Dashboard Direct Cutover
• One way to depersonalize the bad news is to tell the story
using numbers and metrics.
• The test team exists to generate useful information and
effectively communicate that information to key testing
stakeholders. The results reporting process is the method by
 • This approach, in short, the old product or system is shut
down and the new product or system is released. In general, a
target, or release, date is agreed upon and the new product
or system simply replaces the old.
• This approach is also appropriate when releasing a new
product or system, when quick delivery is critical, or when the
existing product or system is so poor that it must be replaced
as soon as possible.
• Direct cutover may also be appropriate when a system is not
mission critical–that is, the system’s failure will not have a
major impact on the organization.
Parallel
• The parallel approach to implementation allows the old and
the new product or system to run concurrently for a time.
• The parallel approach is appropriate when problems or the
failure of the product or system can have a major impact on
the organization.
• Before switching over completely to the new system, the
organization may run both systems concurrently in order to
compare the outputs of both systems. This approach provides Final Project Report
confidence that the new system is functioning and performing
properly before relying on it entirely.
Phased
users of the product the overall project
or system schedule and budget
Project Closure
Five Circumstances for Ending a Project
• Normal: A project that ends normally is one that is completed
as planned
• Premature: Occasionally, a project team may be pushed to
complete a project early even though the product or system
may not include all of the envisioned features or functionality.
• Perpetual: Some projects seem to take on a “life of their own”
and are knows as runaway, or perpetual, projects. These
project never seem to end.
• Failed: Sometimes projects are just unsuccessful. In general, a
project fails if insufficient attention is paid to the people,
processes, or technology.
• Changed Priorities: In some circumstances, a project may be
terminated as a result of change in priorities. Financial or
economic reasons may dictate that resources ar no longer
available.
Issues Near Project Completion
• Team members are concerned about future employment
• Bugs still exist
• Resources are running out
• Documentation attains paramount importance
• Promised delivery dates may not be met
• The players may posses a sense of panic
Reports should include:
Project summary Outstanding issues

Project description Itemized list and expected

completion

Project MOV Any ongoing support

required and duration

Scope, schedule, budget, and Project documentation list

quality objectives

Comparison of planned versus Product or systems

• Following the phased approach, the product or system is
actual documentation
released in modules or in different parts of the organization
incrementally.
Original scope and history of User manuals
• The phased approach may be appropriate when introducing a
any approved changes
software system to different areas of the organizations.
• A phased approach may also allow the project tteam to learn Original scheduled deadline Training materials
from its experiences during the initial implementation so that versus actual completion date
later implementations run more smoothly.
Comparison of Implementation Approaches Original budget versus actual Maintenance
cost of completing the project documentation
Direct Parallel Phased
Cutover Test plans and test results

Implement Provides a safety net Allows for an organized

ation can or backup in case managed approach for
be quick problems are implementing product or
Can be encountered with the system modules in
risky if implementation of the different departments or
product or new product or geographical locations.
system is system Experience with early
not fully Can increase implementation can
tested confidence in the new guide and make later Administrative Closure
Places product or system implementations go • Requirements for administrative closure include:
more when output of old more smoothly – Verifying that all deliverables and open items are complete
pressure on system and new Takes longer and may
– Verifying the project sponsor or customer’s formal acceptance of
the project system is compared. cost more than the
team Takes longer and may direct cutover approach the project
cost more than direct and problems – Organizing and archiving all project deliverables and
cutover approach encountered during documentation
Places more pressure early phases can impact – Planning for the release of all project resources (i.e., project team
on the customer or members, technology, equipment, facilities)
 – Planning for the evaluations and reviews of the project team
members and the project itself
– Closing of all project accounts
– Planning a celebration to mark the end of a (successful) project
Project Evaluation
• There are four types of project evaluations should be
conducted. There should be:
– An individual review of each team member’s performance by
the project manager
– A project close-out review (sometimes called a postmortem
review) by the project manager and project team
– An audit of the project by an objective and respected outside
party
– An evaluation sometime after the product is released or the
system is implemented to determine whether the project
achieved its envisioned MOV.
Individual Performance Review
• Begin with the individual evaluating his/her performance.
• Avoid “why can’t you be more like …?”
• Focus on specific behaviors, not the individual
• Be consistent and fair
• Reviews should provide a consensus on improving
performance
Project Close-Out (Postmortem) Review
• The focus of this review should include the following:
– Review the initial project’s MOV
– Review the project scope, schedule, budget, and quality
objectives
– Review each of the project deliverables
– Review the various project plans and the project and product
methodologies
– How well did the project team perform?
Project Audit
• To provide a more objective view of the project, an audit or review by
an outside party may be beneficial for uncovering problem, issues, or
opportunities for improvement.
• The auditor or audit team should focus on how well the project was
managed and executed. This may include the project plans, processes,
and methodologies. In addition, the auditor or audit team should
assess whether the project manager and team acted in a professional
and ethical manner.
Evaluating Project Success – The MOV
• The MOV, or measureable organization value, was defined at the
beginning of the project. It provided the basis for taking on the project
and supported many of the decision points throughout the project life
cycle. Many of the benefits envisioned by the implemented system
may require and players may require weeks or months before they
are realized.
• Although the different project stakeholders and players may have
different views as to whether the project was a success, it is important
to assess the value that the project provides the organization.
• The evaluation of the project’s MOV may be intimidating–it can be the
moment of truth as to whether the project was really a success.
However, a successful project that brings measureable value to an
organization provides a foundation for organizational success.
Infrastructure Management
• For an organization's information technology, infrastructure
management (IM) is the management of essential operation
components, such as policies, processes, equipment, data, human
resources, and external contacts, for overall effectiveness.
• Among other purposes, infrastructure management seeks to:
– Reduce duplication of effort
– Ensure adherence to standards
– Enhance the flow of information throughout an information
system
– Promote adaptability necessary for a changeable environment
– Ensure interoperability among organizational and external
entities
– Maintain effective change management policies and practices
Categories of Infrastructure Management
Systems Management
• is the management of the information technology systems in an
enterprise. This includes gathering requirements, purchasing
equipment and software, distributing it to where it is to be used,
configuring it, maintaining it with enhancement and service updates,
setting up problem-handling processes, and determining whether
objectives are being met.
Network Management
• is the construction, design, and use of a network, including the
physical (cabling, hub, bridge, switch, router, and so forth), the
selection and use of telecommunication protocol and computer
software for using and managing the network, and the establishment
of operation policies and procedures related to the network.
Storage Management
• is the management that related of the data storage usage in
enterprise. Storage has been divided into primary storage (which
holds data in memory) and secondary storage (which holds data on
hard disks, tapes, and other devices requiring input/output
operations).
Network Configuration Management
• Network configuration management (NCM) is the process of
organizing and maintaining information about all the components of a
computer network. When a network needs repair, modification,
expansion or upgrading, the administrator refers to the network
configuration management database to determine the best course of
action. This database contains the locations and network addresses of
all hardware devices, as well as information about the programs,
versions and updates installed in network computers.
• Network configuration management tools can be vendor-neutral or
vendor-specific. Vendor-neutral tools, by far the more common, are
designed for networks containing hardware and programs from
multiple suppliers. Vendor-specific tools usually work only with the
products of a single company, and can offer enhanced performance in
networks where that vendor dominates.
• Advantages of network configuration management include:
• Streamlining the processes of maintenance, repair, expansion and
upgrading.
• Minimizing configuration errors.
• Minimizing downtime.
• Optimizing network security.
• Ensuring that changes made to a device or system do not
adversely affect other devices or systems.
• Rolling back changes to a previous configuration if results are
unsatisfactory.
• Archiving the details of all network configuration changes.
FCAPS
• FCAPS is a network management framework created by the
International Organization for Standardization (ISO)
• FCAPS categorizes the working objectives of network
management into five levels. The five levels are:
– fault-management (F),
– configuration level (C),
– accounting level (A),
– performance level (P), and
– security level (S).
• At the fault management level, network problems are found
and corrected. Potential future problems are identified and
steps are taken to prevent them from occurring or recurring.
With fault management, the network stays operational, and
downtime is minimized.
• At the configuration management level, network operation is
monitored and controlled. Hardware and programming
changes, including the addition of new equipment and
programs, modification of existing systems, and removal of
obsolete systems and programs, are coordinated. At the C
level, inventory of equipment and programs is kept and
updated regularly.
• The accounting management level, which might also be
called the allocation level, is devoted to distributing resources
optimally and fairly among network subscribers. This makes
 the most effective use of the systems available, minimizing
the cost of operation. The A level is also responsible for
ensuring that users are billed appropriately.
• The performance management level is involved with
managing the overall performance of the network.
Throughput is maximized, network bottlenecks are avoided,
and potential problems are identified. A major part of the
effort is to identify which improvements will yield the greatest
overall performance enhancement.
• At the security management level, the network is protected
against hackers, unauthorized users, and physical or
electronic sabotage. The confidentiality of user information is
maintained where necessary or warranted. Security systems
also allow network administrators to control what each
individual authorized user can (and cannot) do with the
system.
Implementing a Database
• The implementation phase of a database project requires
careful management and there are key practical and strategic
issues to consider.
• By this stage, you’ve already spent time planning, identified
specific requirements, selected a supplier, bought a package,
made customisations and identified any necessary
improvements to your ICT infrastructure. It’s time to
implement your system.
• The key things to remember for a successful implementation
are:
– Take it slowly – rushing leads to error
– Pay attention to detail
– Clean up your data before you add it
– People are your biggest issue (and asset)
– Evaluate what you do
• Before you install the system, make sure your ICT system and
infrastructure (server, network, PCs etc.) is up to scratch. Will
you be able to run all your software at once (databases can be
hungry for computer resources)? Nearly is not good enough.
Once you’re satisfied. It’s time to install it.
– Key point: Good databases perform badly on
poor/old servers/PCs.
• You must have someone in overall day-to-day charge of the
database (often called the Database Manager or Database
Administrator). They are the first point of call for problems,
liaise with the suppliers and keep an eye on issues and
developments. They should also link in to senior management
and recommend decisions on changes and future
expenditure. Appoint them before installation.
• A shared database can be a culture shock. Demonstrate what
difference the database will make to individuals as well as the
organization. Listen to needs and concerns. You don’t want
people working around the system and not using it. Fear of
change is a big issue. Set practical, manageable expectations.
The first few weeks will be time consuming and problematic.
Major benefits take a little while. The biggest challenge is
when a system offers a big improvement to the organization
but may have a negative and demanding impact on specific
users.
• Key point: People are the key to an effective database.
• However good your database is, you need to put data in and
get information out. It takes time and effort and you’ll
probably need a good spring clean of what you have. (If
you’ve never had a database before and don’t intend adding
old data you can skip this bit). List every source of data you
have. Your colleagues probably have an Excel spreadsheet
each, perhaps a few Access databases, an old ‘main’ database
and contacts in Outlook. Compile your list, collate the data
and clean it up. Imagine moving home – you don’t simply
bundle rubbish into boxes and pile it out again. Sort out, clean
up and throw away as you go along. A new database won’t
sort out bad data or tidy up a poor filing system.
• Key point: Only ever add clean data to a new system.
• When doing data migration:
• Decide what data structure you want (you’ll probably have
done this when you bought the system)
• Check your data (you can do this manually or have suppliers
write scripts [small computer programs that match data and
ask you to check problems])
• Throw away duplicates (e.g. someone with the same name
and three different phone numbers they no longer have) and
make sure the data is clean.
• When doing testing:
• For any system, there are two ways to test – use both:
• Scripted or structured testing – you follow a script/process
and perform a key task e.g. adding a contact, writing a report
or managing an event
• Monkey testing – you try your best to break the system and
see what weird and wonderful things it can do (that it
shouldn’t)
• You’ll need to write test reports and have someone to
manage and oversee testing.
• After testing, you may need to redevelop parts of the system
or change your expectations or processes.
• Key point: The time and effort you invest now will pay off
quickly. Better to find the problems before you use it for
‘mission critical’ work.
• Most people in your organization probably get by with simple
software. Don’t assume they can do the same with your
database. You need everyone to use it the same way (more or
less). Investment in training is key to a successful system
because:
• People are happier with the system and are ‘bought in’ to its
success
• People work more effectively with it (and don’t go back to
their spreadsheets)
• Processes are done the ‘organizational’ way - systematically
• People are more efficient and better able to get on with their
job
• Key point: Don’t ever install a new system without training
everyone who needs to use it. The users will rebel and the
system will fail within months.
• When setting standards:
• The more people who use a system, the more you need
standards and agreed processes. It’s important to set these in
advance and make sure everyone is signed up and agreed. A
simple set of principles will make a huge difference to data
quality and performance.
• Databases work best when well set up and maintained and
depend absolutely on quality data. You must agree how to
manage data.
• Key point: Agree standards and principles and make sure
someone senior oversees them.
• ‘Go live’ is the day you let users loose on the system with real
data to use for their day-to-day work. If you can, let one
department get to grips with the database at a time - maybe
phase the implementation over a few days or even weeks.
That way if there are any last minute issues, they can be fixed
before everyone finds them. There comes a point when the
whole organization will change over to the new system. Make
sure you’re ready and have a fallback position in case things
go wrong.
• Key point: Switch over carefully and be ready to (temporarily)
go back to the old way of working if there’s a serious problem.
• Just as an ICT system should have a support contract, any
serious investment in database technology will need expert
help, almost always from the supplier (they built it, they know
best how to fix it). This usually comes in the form of a joint
licence/support contract (or online help). It can be expensive
(from 10 to 20% of the system cost) but is worth it. If it
breaks, you may end up with nothing.
• You should also consider:
 • A reporting (change control) process – what changes are – Platform as a service (PaaS). Platforms that can be used to
wanted or needed and why? Are they improvements or bug develop and deploy applications.
fixes? – Software as a service (SaaS). Software deployed as a hosted
• Monitoring and evaluating how the system is working - what service and accessed over the Internet.
could be improved? Do people and processes need to adapt? • Four key operational requirements that organizations must
• Identifying changes (opportunities) to working practices in the meet to operate a private cloud. Today’s technology
organization and unexpected issues management teams must:
• Longer term support costs for the system to be most effective – have standardized operating procedures for the most
• Maintaining your database champion and project steering common tasks in their virtualized environments, such as
group to support the project provisioning,
Virtualization – patching, monitoring, and evacuating workloads from the
• The promise of portability, ease of provisioning, and overall cost of pool;
management make virtualization attractive to most enterprise IT – hand off these standard functions to automation software to
organizations. With the current technologies available, virtualization drive up the efficiency and predictability of these tasks;
is occurring both inside the IT organization as well as at outsourced – provide a means of self-service to their developers so that the
hosting providers. most common requests can be fulfilled rapidly and with as
• Virtualization provides a better business value than traditional little human intervention as necessary
methods. Most businesses are migrating key servers and – be prepared to share the common cloud infrastructure across
applications to virtualized environments and these businesses aren’t business units and internal departments, or risk driving down
stopping at one or two of their servers, but are virtualizing the vast utilization and thwarting any ROI.
majority of them.
• Virtualization is more than just a trend, it’s an integral part of the IT
organization’s infrastructure. While virtualization offers
organizations the promise of tremendous agility and flexibility at
reduced costs, it also includes complexities and requires new ways
of monitoring and managing environments.
Virtualization Management
• Virtualization Management covers all aspects of managing a
modern virtual or software defined data center. This includes
managing across virtualization platforms and clouds,
monitoring the performance and availability of the
virtualization platforms (hypervisors) and the clouds,
monitoring the capacity of the virtualization platforms and
clouds, monitoring the performance of the applications
running on these platforms and clouds, automatically
provisioning these environments, securing these Private Cloud Architecture
environments, and ensuring that the data in these
environments is always protected and available.
• Activities related to virtual infrastructure implementation:
• Network Configuration
• Storage Configuration
• Logical Partitioning Configuration
• Operating System (OS) Installation
• System Management Software (existing and future
versions of software)
• Knowledge Transfer
Begin to Cloud …
• The expected benefits of enterprise private cloud include:
– Increased agility, including significantly reduced
provisioning times.
– Greater efficiency, including energy savings, due to
better resource utilization.
– High availability with minimal incremental cost, by
taking advantage of enhancements to industry-
standard hardware and software.
– Improved capacity management, taking advantage
of new business intelligence tools.
• Several attributes that distinguish cloud computing from
conventional computing. These attributes are:
– On-demand self-service
– Broad network access
– Resource pooling
– Rapid elasticity
– Measured service
– Sharing by multiple tenants
• There are three primary categories of cloud computing
service:
– Infrastructure as a service (IaaS). Computing infrastructure,
such as servers, storage, and network, delivered as a cloud
service, typically through virtualization.