Analytic Modeling Techniques for

Predicting Batch Window Elapsed Time

Debbie Sheetz, BMC Software

Techniques for capacity planning/modeling were first developed in order

to predict system responsiveness for interactive work. But many
companies have critical work which runs periodically (daily, monthly), non-
interactively, and with an elapsed time requirement (“batch window”).
Elapsed time prediction utilizes different modeling and analysis
techniques than interactive workload analysis does. These techniques are
outlined and applied in four case studies (UNIX/Windows systems): (1)
one job (2) multiple simultaneous jobs (3) a sequence of smaller jobs (4)
hundreds of sequences of smaller jobs.

1. INTRODUCTION
The original motivation for developing capacity
planning tools was the advent of “interactive”
computing, where the customer was a person, and
they cared how long it took for their “transaction” to
process. Prior to that, most computing consisted of
various varieties of batch processing and the
customers were internal, so not much was done in
the way of capacity planning. So the first analytic
models had three types of workloads available:
Interactive Time Sharing (TS), Interactive
Transaction Processing (TP), and Batch Processing
(BP). Usually analysis and modeling was focused
on the peak of the interactive workload(s), and batch
was just a low-priority stream of work that utilized
the resources not required for the interactive
processing. The overnight period was then used
exclusively for “production” batch and the
“interactive” day would begin again the next
morning. All the while, there continued to be
particular batch work whose completion was critical
to the business and whose precise time of
completion was known. Examples of this type of
work would be reconciliation of financial accounts,
database updates, or billing cycles.
The key similarity between interactive and critical
batch work is that there is a very specific
performance expectation, e.g. “under .1 second
response time” or “finish by 8 AM”. The key
difference between interactive and batch work is that
interactive work is typically evaluated one
transaction at a time where critical batch is typically
a very large number of transactions which are
submitted for processing all at once, and the
completion of the last request is the only timing that
is of interest.
It is important to note that examples of critical batch
exist on all platforms – it is the oldest use of
computing systems and continues to this day. On a
PC you have overnight backups and virus scans and
on large mainframes and distributed systems you
have overnight database updates and multi-day
sequences to produce bills. With all critical batch,
there is always an explicit scheduling and frequency;
for example, you might schedule a backup to be
performed daily, at midnight. Billing cycles are
usually executed monthly. The scheduling is
frequently also explicit about what state the server
will be in, i.e. not running any other work, for two
reasons:
(1) There may be a requirement that everything
else be “finished”. For example, open files
cannot be backed up or nightly account
reconciliation cannot occur if customers are
still entering additional transactions.
(2) Having all computing resources available for
a compute-intensive activity always favors
the best possible performance!
For the largest examples of critical batch work, one
or more large servers may be required to perform
the necessary processing, and these servers are
likely dedicated to this processing. The timing is
usually dictated by business requirements rather
than by server availability. For smaller examples,
such as a PC backup or virus scan, you intentionally
schedule these activities outside of your “interactive”
use of the PC, e.g. 3 AM, expecting the work to be
completed before you need to use the PC
interactively again.
It is also common to have explicit expectations
about when the batch activity can start as well as
the time by which it must be complete, for example,
account reconciliation at a financial institution cannot
start until the “interactive” day has completed, and
the reconciliation must be completed before the next
“interactive” day begins. And this is exactly why
capacity planning analysis is almost always required
–
the amount of work to be completed and the time
available for completion often conflict with each
other.
This is where our work begins.
2. METHODS
So how can the performance demands of batch be
represented? How does this differ from
characterizing interactive work?
2.1 Study Time Period Selection
One of the most striking differences between batch
and interactive work is the length of time selected for
analysis.
Most commonly interactive work peaks at one or two
times during the day, so likely selections range from
a one hour peak to a five minute peak. The amount
of time selected depends only on how SLAs are
defined: if the business will spend money to
guarantee performance as measured for an hour,
than a one hour peak should be selected; if the
business will spend money to guarantee
performance for the highest activity for five minutes,
then a five minute peak should be selected. Peaks
may vary by day of week, time of month, or time of
year, and that needs to be taken into account when
selecting which day(s) to select the peak periods
from.
Batch work runs until it is done, e.g. 1 hour, 8 hours,
3 days, so the selected time period needs to match
the specific characteristics of the batch work being
studied. Depending on the frequency of the batch
work, e.g. daily, weekly, monthly, particular day(s)
will be chosen for study. More discussion of how to
make these selections will be presented in the case
studies, particularly how to deconstruct what
appears as one batch sequence into component
sections as required.
What is completely similar for all performance
analysis is that historical data is of the utmost
importance as it enables analysis of
(1) performance results trends, e.g. that the
batch run time is lengthening
(2) workload trends, e.g. the amount of work
load is increasing
Objectively it doesn’t matter what the trend is or isn’t
– what matters is that the analyst can confirm what
has been stated as the current performance results
and changes (if any) in the workload.
Also historical measurement data allows the
selection of multiple “baseline” study periods – no
performance analysis should be attempted using
only one selected time period, or at least such
results should be considered tentative at best!
2.2 Resource Performance Objectives
Another striking difference between batch and
interactive work is the interpretation of server
resource utilization.
For interactive work, there is endless discussion of
what the proper settings are for resource thresholds:
CPU queue length less than 2 per processor, 80%
CPU utilization, 40% disk utilization, etc. All of these
thresholds are trying to capture what level of delays
due to queueing will degrade performance too much.
Each interactive transaction contributes a tiny
amount to overall utilization – the fact that there are
hundreds of simultaneous users causes the
aggregate higher utilizations. Also the simultaneity
isn’t coordinated – no matter what your favorite
distribution of arrivals (e.g. random, hyper-
exponential, etc.), no one is going to argue that the
loading is even. The summary of the interactive
technique is that capacity planning seeks to keep
the aggregate utilizations and resulting queueing
below the level at which response time becomes
unacceptable.
Batch analysis couldn’t be more different: maximum
throughput/minimum elapsed time is achieved when
one or more resources achieve 100% utilization.
The first reaction to this principle is this has to be
wrong – think of the queueing!! But the key insight
here is that there is usually no queueing at server
resources because the queueing is occurring inside
the batch processing, i.e. another transaction is not
presented to the server to process until the last one
has already completed. So expect 100% utilization
with a zero queue – that means that the processing
using current resources is optimal. If more than one
resource is measured at 100% utilization, even
better! And if you don’t see at least one server
resource (CPU or disk) at 100%, that demonstrates
that there is a constraint in either the application
design or software implementation that is preventing
the batch stream from achieving optimal
performance.
2.3 Characterizing the Work
The biggest difference between representing
interactive and batch work is the source: multiple
users issuing uncoordinated requests vs. a stream
of requests occurring precisely one after the other.
Also, it’s important to represent the difference
between multiple simultaneous requests vs. a single
(or multiple) stream(s) of requests.
The resource demands are not represented
differently, i.e. a “transaction” requires a certain
number of milliseconds of CPU, a certain amount of
I/O, etc. How you choose to define a transaction
can be different, but after that choice has been
made, the results have the same form.
Another practical difference is that interactive work
is usually characterized by aggregating the activities
of many users where batch work often focuses on a
very small number of processes, sometimes only
one. And even when there are multiple batch
processes, they are usually represented separately
rather than being aggregated.
2.4 Analysis Techniques
The most common approach for predicting
interactive performance is analytic modeling; batch
performance analysis begins with a bit of arithmetic,
frequently supplemented by analytic modeling.
Building a suitable analytic model depends on
having adequate system measurement data
available, i.e. both system resource measurements
as well as detailed process data representing the
application components. As introduced above,
elapsed time analysis typically indicates selecting all
of the elapsed time, or as will be shown in the case
studies, all of the time within a phase of the batch
processing. The process data is used to isolate the
process(es) of interest into model-ready
“workload/transactions”.
2.4.1 Applying Business Volume to the
Analysis
The standard model is then supplemented using two
types of additional information:
(1) total job/process elapsed time (referred to
as BATCH-RESPONSE)
(2) total number of business requests
processed during entire execution (referred
to as BATCH-TRANSACTIONS)
Four categories of changes may now be evaluated
with a model and a calculator:
(1) Application change (e.g. more I/Os, I/O to
different devices, change in CPU
requirements, fewer physical I/Os)
(2) Competition from other workloads (e.g. new
workload(s), changed workload(s),
workload(s) removed)
(3) Hardware change (e.g. new CPU, additional
CPUs, new devices, new caching)
(4) Change in application volume
The first three will be expressed as changes in
various model parameters (using standard analytic
model “what-if” procedures). The fourth case,
change in volume, requires no change to the
model.
After the necessary changes have been applied to
the model and the model re-evaluated, the new
throughput and response time will be calculated by
the model.
In order to determine the new job response time (for
all cases):
New BATCH-RESPONSE =
New BATCH-TRANSACTIONS *
New BATCH-RESPONSE TIME (2.4.1)
When only the application volume is changing, no
model evaluation is required: the New BATCH-
RESPONSE TIME is the baseline response time,
thus, only the new volume is needed to calculate the
result.
For cases where transaction volume is not
changing, New BATCH-TRANSACTIONS is the
same as Baseline BATCH-TRANSACTIONS.
2.4.2 Applying Stream Constraints to the
Analysis
Another change applied to the standard model is
representation of a stream of transactions vs.
interactive arrivals of transactions. This is effected
by augmenting the transaction description with the
maximum possible concurrency, e.g. for a single
job/process without threads, the maximum
concurrency is 1. Also, calibration of such a
transaction requires an addition to the standard
model calibration procedures: Check the result of
multiplying the transaction throughput and the
transaction response time. Ideally, this should come
out to about 3600 seconds (1 hour). If it comes out
to less than 3600, some other processing is missing,
and should be added. If it comes out to more than
3600, you would have received notice of saturation
during model evaluation and need to consider what
might be removed from the current transaction
processing requirements. If the maximum
concurrency is 2, you use 7200 seconds as the
standard of comparison.
Although the examples presented use distributed
systems data (Unix and Windows), the principles
and techniques apply to any computing system.
These techniques were first developed for
mainframes, then enhanced for mid-range systems,
then applied to distributed systems.

An important observation here is that not every

analysis includes using an analytic model, but the
process of interpreting and preparing the data for a
model provides the necessary context for an
effective analysis.

3. RESULTS
Figure 1. Server CPU Utilization
(possible 400%)
Four case studies will be presented, demonstrating
various aspects of the analysis techniques described
above. As suggested in the introduction, there is
both great variety in particular examples of elapsed
time analysis as well as common principles applied
to every one.

Each case study reflects consulting work done for

one phase of performance analysis, so “final”
conclusions for each case study are unknown. The
intended contributions were to identify the most
important characteristics and factors relating to
capacity planning solutions, if there was a capacity
planning solution at all. And even though there is
variety amongst these cases, these examples Figure 2. Physical Processsor Utilization
certainly don’t cover all types of batch analysis!
The job runs within Sybase (which is represented by
3.1 Daily Batch Job Analysis Case the “dataserver” processes) and executes from
about 7:00 to 7:30 AM. What’s notable here is that
The first case study was a request to analyze why a not only is the system utilization well below 400%
particular batch job took so long to run and thus how (400% = 4 processors) and no individual processor
it could be made to run faster. The staff above 75%, the overall system utilization actually
performance analysts had suggested allocating decreases when the job of interest is running! The
additional CPUs to the job as well as increasing the dataserver processes do record increased utilization
amount of memory available. The system had 4 (Figure 3):
processors configured. Two weeks of measurement
data were obtained to study the characteristics of
the job.

The focus here is to find what the constraint is to the

job running faster, and given the server data we
have, checking server resources is the first step.

We are hoping to find either CPU or disk resources

at 100% utilization, which would indicate that the job
is making maximum utilization of the available
resources, and our only task is to identify what kind
of hardware upgrade will produce the most
improvement in performance.
Figure 3. CPU Utilization by Process
First check on the CPU resource utilization, Name
checking both the overall system utilization as well
as the per physical processor utilization (Figure 1, Maybe the dataserver processes are individually
Figure 2): constrained?

Figure 4. CPU Utilization per Sybase
dataserver Process
No 100% utilization here (Figure 4).
Similar checks are made for I/O (maximum disk
utilization is 4% and CPU wait for I/O is 2% and
under), paging (close to zero), and network (all not
shown due to paper length requirements). All of this
indicates that hardware constraints of any type are
unlikely to be the cause for slowness so instead it
must be the application implementation itself.
Another possibility is that there are some hardware
constraints, but they don’t appear using 15-minute
average resource reporting.
For the next phase of the study, measurement data
is instead summarized for each minute (instead of
for every 15 minutes), and the resource utilization is
higher than what was seen before, but nothing is
near to 100%. For example, it is still evident that the
three Sybase processes are not saturated and in
fact during the heart of the processing, two of the
three have no load at all (Figure 5):
Figure 5. CPU Utilization per Sybase
dataserver Process
The CPU Idle due to I/O is as high as 10%, disk
utilization at 20%, paging peaks at 15 pages/sec –
still no indication that any server resource is over
used.
The final step in this analysis is to look at the relative
use, i.e. “service time”, for CPU and I/O, since from
a hardware perspective that is the only response
time component that can be reduced (response time
= service time + queueing time). CPU service time
dominates, so the only upgrade that would offer a
speed up is a faster CPU. Note that additional
CPUs offer no relief since there is no measured
CPU queueing and the primary portion of the job is a
single process using its one processor. The most
important question to be answered is why is the
application structure the way it is and what is the
true constraint within the application which limits
performance? We have shown that it’s not the
server resources! This is definitely not a capacity
planning problem.
Although it’s not shown in the paper due to space
constraints, the two weeks of measurement data
was overwhelmingly consistent, so representative
single samples are shown above.
3.2 Monthly Batch Processing Analysis Case
The second case study was a request to analyze
why a particular batch cycle took so long to run and
to predict how it would perform when the workload
was increased. It is a billing application and the
performance requirement is that when the workload
is doubled, the billing cycle still needs to complete
within nine days. The system had 12 processors
configured and the application uses Sybase for the
database backend. Two examples of this billing
cycle were measured in order to study the
characteristics of the cycle: the March cycle
processed 1.7 million accounts and April processed
1.825 million accounts.
The billing cycle is completing within the required 9
days now, but without much time to spare. We
begin the analysis by selecting the longest phase of
the billing cycle, which currently requires 1 ¼ days
to complete, and applying the prediction formula
(2.4.1), is projected to take 2 ½ days to complete,
enough to violate the performance objective.
How is this billing application structured? This
sample is taken from a day when the billing cycle is
executing continuously from midnight to around 6
PM (Figure 6):
 Figure 6. CPU Utilization by Process
Sybase is configured with 6 “dataserver” processes
(which enables it to use 6 out of the 12 processors
configured on the system), and the customer
accounts are extracted by 5 “extract98” processes,
each of which extracts accounts from one database.
Based on this profile, the workload will be
characterized by isolating each of the dataserver
processes, and each of the extract processes.
Another interesting piece of data is that even though
the number of accounts increased by 7% between
March and June, the elapsed time increased by
30%. This is even worse than anticipated, so we
know significant performance degradation is already
occurring.
We are hoping to find either CPU or disk resources
at 100% utilization, which would indicate that the
application is making maximum utilization of the
available resources, and thus indicate the
appropriate hardware upgrade. Figure 6 shows that
system CPU utilization is at about 6 to 7 processors
out of 12. So there is ample available CPU
capacity. Disk utilization is as high as 80% -- no
saturation here, either. So the main question has to
be why the application is not making full use of the
available CPU resource.
Figure 7. CPU Utilization per Analysis
Workload
Figure 7 shows two runs of the database extract
phase, and we can clearly see the important
characteristics here:
(1) the database extracts are quite evenly
balanced (good) and
(2) the Sybase dataservers are not as evenly
matched (not so good)
(3) some dataservers are using just about
100% CPU, i.e. one processor (good).
The second run completes more quickly than the
first, and most likely it is due to the much more even
loading of the dataservers, all showing about 100%
CPU utilization.
So the most immediate “upgrade” would be to
configure more dataservers in order to access the
available CPU capacity. Also, each time the cycle is
being run, less than even utilization of the
dataservers most likely indicates a constraint other
than hardware inhibiting maximum server
performance. Uneven loading also appears to
result in uneven loading of the extract processes,
too, resulting in a longer runtime (Figure 8).
 Figure 9. Process CPU Utilization Profile

Figure 8. CPU Utilization by extract
Process
An analytic model predicts that about 75% of the
growth could be absorbed with the current
configuration, assuming that even loading of the
dataservers (and extracts) is possible and
sustainable.
Details of the process start times are observed,
allowing division of the sequence into five phases,
corresponding to the “FE_EMC_PRO” process
running at the time: “A” from 20:31, “B” from 21:08,
“C” from 22.41, “D” from 23:37, and “E” from 00:22
(Figure 10).

Similar analysis would be required for each of the

additional segments of the billing cycle as different
recommendations could result.

More detailed application analysis might reveal

application tuning opportunities, but most companies
choose adding hardware as being more cost-
effective than actually trying to alter the application
performance directly. Figure 10. Listing of selected process
Start Times
3.3 Batch Processing Sequence Analysis Case

The third case study analyzes a stream of batch An analytic model is constructed for each phase,
processes which currently complete in 4 hours and labelling the main transaction as “EMC-A”, “EMC-B”,
predicts what system upgrade(s) would be needed etc. for easy identification. The BATCH-
to maintain that elapsed time when the workload is TRANSACTIONS for each phase is 20,000
doubled. The system has 8 processors configured (preparing for use in equation 2.4.1).
and the application uses Oracle for the database
backend. The model for “EMC-B” is used as an example. The
original process run time is about 1.7 hours. The
The sequence begins in the evening and ends early model is evaluated (Figure 11).
the next morning. Measurement data for the two
days is analyzed in order to see the detailed process
structure. Specifically, there is a sequence of
processes named “FE_EMC_PRO” which generate
the activity (Figure 9).

Figure 11. Analytic Model Transaction

Report

To check calibration for the transaction of interest,

11.44K transactions * .30 sec/transaction = 3434
seconds. Compared to 3600 seconds (because
throughput is expressed as transactions per hour),
that’s 95%, or 5% from perfect calibration. This is
acceptable as is.
Now for the upgrade analysis. We observe the
transaction profile, i.e. how much time is service,
and how much is wait:
This transaction is almost entirely made up of CPU
service time. This immediately indicates that the
only type of upgrade that will have a significant
effect on performance will be one where the
individual processor is much faster than the current
processor. Adding additional processors will not
have any effect (because there’s no CPU Wait time)
and I/O upgrades will have very little effect because
not much time is spent doing I/O.
The proposed upgrade is from current Sun 15-
processor system to an 8-processor Sun system
where the individual processor is about 9 times
faster. We “install” the upgrade on the system, and
reevaluate the model:
The relative response time of .29 indicates that we
will easily be able to double the transaction volume.
Also note that a substantial CPU upgrade makes the
I/O relatively more important to performance in the
future.
The transaction volume is doubled, and the model is
re-evaluated (Figure 12):
Figure 12. Analytic Model Transaction
Results
So, if the new desired volume is 40000 transactions,
and the new response time is .09 seconds, the total
job run time is 1 hour, which is actually better than is
required.
This is repeated for each of the 5 models, and in
every case, the upgrade is adequate.
The summary of the baseline performance is that
the transaction profiles are dominated by CPU
service time in each of the first four phases. These
four phases account for about 4 hours of
processing. The fifth phase, which is a clean up and
FTP phase, uses I/O more than CPU, but is a very
short phase (about 20 minutes).
The baseline job stream response time is
approximately 4.3 hours:
20000 transactions *
(.09 + .31 + .13 + .14 + .11) sec/tran = 4.3 hours
Figure 13. Baseline Model Response Time by
Phase
The summary of the ‘what-if’ modeling result is that
the relative response times for the first four phases
show that the planned CPU upgrade is adequate,
i.e. a relative response time of .5 or less indicates
that double the business volume can be handled
without extending run time. The fifth phase, does
not show the same improvement, only .70 of
original, so that phase will run a bit longer than the
original (40000 transactions * .08 sec/tran = 53
minutes).
 There are hundreds of job sequences to be
performed, but no single sequence occupies the
entire batch window – generally individual
sequences are around an hour in length, beginning
with a data acquisition phase, then followed by data
analysis, database update, then a cleanup phase.
The set of sequences begins at midnight and ends
the next morning. Measurement data was obtained
for two weeks (14 samples of daily activity), and one
day is shown as an example here.

First, an overall view of the AIX frame ():

Figure 13. 'What-if' Model Response Time by

Phase
3.4 Batch Processing with a Large Number of
Jobs Analysis Case
Figure 15. Frame CPU Utilization by
The fourth case study analyzes a stream of batch
processes which currently complete in 8 hours and SPLPAR
predicts what system upgrade(s) would be needed
to maintain that elapsed time when the workload is The sequence of data acquisition, processing, and
quadrupled. The system has 16 processors cleanup is shown in pink, and the backend database
configured and the application uses Oracle for the activity in blue (
database backend. The system is an AIX Power6
frame, with 4 SPLPARs processing particular
portions of the workload (2 application partitions, 2
database partitions). The workload consists of
thousands of jobs.

Figure 146. Workload CPU Utilization

When the individual jobs/processes are examined,

the underlying application structure becomes evident
– many, many small jobs are used (Figure 15).
Given the relatively short sequences and the
number of pieces the work is divided amongst, the
sequencing aspect does not require explicit
representation, unlike the third case study. deployment to use the available CPU capacity for
the first few hours is unlikely. More promising is to
examine the cause(s) for the “tail” of the utilization,
from 4 to 8 AM. If the “tail” could be eliminated, the
batch processing could end 2 hours earlier (Table
1), which creates 25% additional capacity. So
instead of needing 64 processors for the increased
workload, only 48 will be needed.

Time Current Current Ideal

AM Processor Available Processor
s Used Processor s Used
s
4:00 13.4 2.6 16.0
4:30 10.6 5.4 16.0
5:00 7.3 8.7 16.0
5:30 5.0 11.0 2.3
Figure 157. Number of processes by
6:00 4.6 11.4 --
process name
6:30 4.9 11.1 --
7:00 2.9 13.1 --
The ‘what-if’ analysis is fairly straightforward: if no 7:30 1.7 14.4 --
change is made to the application, since the AIX Tota 50.3 50.3
frame is already at 100% utilization, 4 times the l
workload will require 4 times the number of
processors, e.g. 64 instead of the existing 16 (see Table 1. 'What-if' the processing tail
section 2.4.1 above). However, the measurement was eliminated
data shows opportunities to improve the throughput
if the existing CPU capacity can be more fully
utilized, i.e. the ideal batch CPU utilization of 100% 4. CONCLUSIONS
is being achieved for about only 3 of the 8 hours
available (Figure 16): Analytic models and their associated techniques can
be applied to elapsed time analysis as well as to
traditional interactive workload analysis. Maximum
performance is characterized by server resource
utilization of 100%, completely the opposite of
what’s desirable for interactive workloads. Selected
measurement periods usually encompass the entire
batch processing window. Variation in processing
profiles indicates the need to break the processing
up into phases, each of which is analyzed/modeled
separately. Many examples of batch processing are
in fact not constrained by server resources and
coming to that conclusion is the single most
important step in moving forward to identify what
about the application or its implementation is in fact
the real constraint. Lack of accurate elapsed time
Figure 168. Frame Processors Used analysis causes many poorly performing
applications to continue in that condition for longer
Another way to think about this is that the green than necessary as well as having both human and
area represents available capacity. Considering that server resources expended without achieving the
the first phase is data acquisition, tuning the desired effect!