Data Gathering

Preparation
The measurements that are required, entail a collection of various metrics which are
collected from a number of different sources.
In order to ensure that the relationships between the different measurements captured
are maintained, all time clocks on the various
application server and client components must be synchronised. The etwork Time
Protocol !TP" can be used to effect such a
synchronisation between servers and clients. #hould synchronisation not be
practically possible, application components which
cannot be synchronised should be e$plicitly and clearly identified, along with the time
differences encountered !in % second increments".
&ther preparation activities include the collection of information which will assist in
the workload characterisation process.
This entails a list of the individual transactions which will be used by the application
in a production environment.
'ttention should be given to high impact transactions, namely transactions which
have the following characteristics(
) high response times
) high server resource impact !transactions which effect data input"
) high numbers of users e$ecuting specific transactions
) high transaction e$ecution rates with regards to specific times of the day
*apture engineers should also have an in+depth understanding of server component
configurations !specific *P,s etc."
and geographic user distributions in the test and the production environments.
#ystem *onfiguration
In order to ensure the accuracy of measurements with regard to virtualised
environments !for capture of prediction purposes",
it is necessary that the virtualisation technology be sub-ect to limitations(

'ttempt to obtain measurements in a non+virtualised environment. If a non+
virtualised environment is not possible,
then the .irtualisation overhead must be established with regards to the specific
situation. This can be established
with the comparison of transaction measurements captured in a virtualised and non+
virtualised environment
/nsure that #0T !simultaneous multi+threading" is switched off on the application
servers
&nly physical processor configurations should be used, 1ogical processors are to be
avoided
The application under test should operate within a virtualised environment which is
isolated from the impacts of
other virtualised applications. Isolation should occur at the processor, disk and
memory level
The number of *P,s assigned should be fi$ed and dedicated to the application only
1ogical disks should operate on dedicated physical disks
Dedicated memory should be made available to the application under test
If running on a virtualised environment, it would be preferable to ensure that other
applications are not running simultaneously,
to keep unwanted network impact activity to a minimum
&ne, 0ulti 2 #tacked ,ser 0easurements
#ingle ,ser 0easurements
3or the purposes of definition,
) an application is composed of a number of business cases which directly relate to
specific business ob-ectives which need to be accomplished by the application, for
e$ample, the placement of a customer order.
) /ach business case is composed of a number of transaction types which relate to
specific activities which must be completed by a user to perform the business
ob-ective. #uch actions may include(
) #tart the application
) 1og &n
) avigate to &rders Page
) ew &rder
) #earch *ustomers
) /nter &rder Details
) *ommit &rder
) 1og &ff
The single user measurement procedures are aimed at the collection of metrics for
each transaction type, which will be used to establish the profiles of transaction types
of the application under investigation. The transaction type profiles are an important
input for the simulation model. The measurements collected should also include the
single user transaction type response times which are important as they are needed to
validate the simulation model.
4ith single user transaction measurements, transaction types are e$ecuted in a single
serial manner !with long pauses between each transaction type e$ecution" against an
unutilised application infrastructure. This requires that the engineer collecting the
measurements have e$clusive access to the test environment. 4hilst transactions are
being e$ecuted against the test environment, various capacity elements of the
application server infrastructure and the network will be utilised.
These metrics will be collected in such a manner as to establish a relationship
between(
) the transaction types e$ecuted,
) the response times for each seperate transaction type e$ecution,
) the network traffic generated during the transaction type e$ecutions, and,
) the server capacity utilisation metrics encountered with each transaction type
e$ecution.
The final aim is therefore, on a transaction type e$ecution basis, to establish the
specific infrastructure service demands
incurred by the application, and the contribution that specific server capacity elements
had to the overall response time for
that e$ecution. This requires that server component metrics be collected at the
smallest possible time interval, namely % second.
4hilst it is ideal that e$isting test scripts !functional or non+functional, eg.
1oadrunner" be used to e$ecute the transaction
types, it is possible to effect the transaction type e$ecutions manually.
5 +' common misunderstanding e$ists that the single user, single transaction type
e$ecutions mentioned are sufficient to understand
the performance of the application under various loads. This is not the case. 4hilst
they do allow for an understanding of the performance
impacts of specific transactions, analysis of multi+user, multi+transaction type loads
requires the use of the analytical m5race model.
The purpose of the single user, single+transaction type e$ecution is to allow for the
collection of specific metrics which are then used
to populate the m5race analytical model. &nce populated with the basic transaction
type data, the performance of the application can then
be established under a wide variety of application transaction loads.

The #tacked #ingle ,ser 0easurement
#tacked single user measurements are used where the impacts, due to a specific
transaction type, are so small that they become difficult
to establish. The transaction types are therefore 6stacked7 either vertically or
hori8ontally and e$ecuted as a transaction type group.
The transaction groups are e$ecuted in a similar manner to the #ingle ,ser
measurement method, by e$ecuting transaction type groups in a
serial manner with a long pause between group e$ecutions.
The transaction type groupings can be structured in a number of ways, depending on
the specific circumstances encountered(
) ' single user can e$ecute mutliple transaction types as a group, without pauses or
with small pauses
) 0ultiple users can e$ecute single transaction types as a group, with or without
pauses
) 0ultiple users can e$ecute multiple transaction types as a group, with or without
pauses
During the parsing process, the total resource impact is evenly distributed amongst the
number of transaction types e$ecuted, allowing for
the calculated average #ingle ,ser 0easurement result. The response time of the
#ingle ,ser measurement is established using the average
response time of all e$ecuted instances of the transaction type.

0ulti ,ser 0easurements
The 0ulti ,ser measurements deliver application platform resource utilisation and
transaction type response time information, with greater
user and transaction volumes. These measurements are required to perform the
second model validation process. .alidation is required in
order to ensure the accuracy and reliability of the m5race simulation modelling
results.
5+ It is usually required that a transaction coincidence matri$ be completed with
every measurement e$ercise. This requires an identification
of the e$ecution paths of transactions in order to understand which server tiers have
been active during the processing of a specific transaction.
4hilst it is e$pected that application developers understand their application
mechanics and therefore know the transaction e$ecution paths of
specific transactions, this is not usually the case in practice. 3urther, it is important to
know which transactions were processed by which
system interfaces, and this is also normally difficult to establish through the
developers. If the measurement plan has been well e$ecuted,
and the measurement functionality allows, the coincidenc matri$ can be established
with reasonable accuracy following an analysis of the measurements.
'lso note that the only guaranteed means by which the coincidenc matri$ can be
established, with absolute certainty, is through an analysis of the
entire network traffic trace of the application during the e$ecution of the transaction
types.
oise 0easurements
/nvironment 6noise7, is the resource utilisation overhead that occurs on the servers in
the production environment, that is not
directly attributable to the active elements of the application under test. /nvironment
noise is usually composed of application
and environment management overhead activity due to other applications or the
application under test. 4hilst the application
6noise7 of the application under test can be readily established in the test
environment, application noise of other applications
operating in the production environment are more difficult to establish, but can be
measured over a period of several months of data.
oise Determination
This is principally determined from data collected in the production environment. The
following production environment data is
required from all servers in the delivery chain(
+ *P, utilisation
+ Disk transfers
+ umber of packets
+ Process information !9*P,, memory and :threads"
It is preferable that these measurements be collected over a period of at least one
month.
0easure #ingle and 0ulti ,sers
The measurement scenarios are always initiated by establishing the time differences
between the clocks of the various server platforms
and the load generation P*s. The etwork Time Protocol !TP" is usually used to
synchronise all servers and P*s in the environment
where the measurements are collected. This is not always practically possible.
The engineer controlling the 1oad;unner testing environment is responsible for
establishing any time differences which are encountered
in practice. 'ny time differences encountered should be determined with reference to
the load generation P*s and these differences
should be identified in terms of a degree of accuracy of % second. The parsing process
can usually accomodate degrees of accuracy of
< seconds, but greater deviations usually lead to a dramatic increase in the amount of
time required to complete the parsing process.
It goes without saying, that if the load generator clocks are out of synch, the best
practice would be to ensure that they are synchronised
with the rest of the environment.
#ingle ,ser 0easurements
These measurements should be collected under e$clusive access conditions, namely
that no other application activity should be taking place
on the servers whilst the transaction types are being e$ecuted. If the application under
test uses batch processing transactions, these
should be scheduled to occur at intervals which do not coincide with the single user
measurement activities. If possible, all batch activities
on the application under test should be temporarily halted whilst the single user
meeasurements are taking place.
In clustered environments, it is recommended that only a single server node be active
during transaction type e$ecution and measurement activities.
4hilst conducting the single user measurements, the collection engineer should
ensure !if possible" that the transaction types required for a
test or business case, be stipulated in a single script, in the order required for the
successful e$ecution of the entire business case.
The script is to be e$ecuted si$ !=" times in a repetitive manner, with ten !%>" second
user sleep time intervals between script iterations.
Transaction types within a script should be seperated by fi$ed user sleep times of five
!?" seconds.
The ? and %> second sleep time spacings are required to facilitate the parsing process,
in that transaction activites can be clearly identified
and relationships established between transaction e$ecution and the resource
utilisation measurements obtained from the various application server
platforms. It is recommended that inter+transaction sleep times be limited to a
ma$imum of ? seconds in order to ensure that total measurement
durations and data si8es remain as small as possible ote that e$tensive and detailed
measurements are gathered at % second intervals during
the measurement collection process.
#tacked single user measurements are similar to single user measurements, with the
e$ception that instead of e$ecuting single transaction types,
transaction type groups are e$ecuted. /ach transaction type group is composed of %>
transactions of the same transaction type which are e$ecuted
in a serial manner, one after the other with no user sleep times.
0ulti ,ser 0easurements
In a similar manner to single user measurements, multi user measurements require
e$clusive access to the test environment whilst e$ecution is taking place.
In clustered environments, it is recommended that all cluster server nodes be made
active. *are should be taken to ensure that load balancing components
are functioning correctly in order to ensure that the application cluster nodes
e$perience balanced loads. This is not usually the case in practice.
The multi user measurement scenario provides for a gradual stepped increase in
transaction volumes and may contain a number of load 6steps7.
The steps may be composed as follows(
+ #tep % @ ? users e$ecuting transaction scripts for %> minutes.
+ #tep < @ %> users e$ecuting transaction scripts for %> minutes.
+ #tep A @ %? users e$ecuting transaction scripts for %> minutes.
+ #tep B @ <> users e$ecuting transaction scripts for %> minutes.
The stepped process can increase after every %> minute period. The %> minute period
is required to ensure reasonably stable transaction response times
and server resource utilisation metrics. The transaction response times and server
resource utilisation metrics for each %> minute 6step7 is used
to validate the m5race model at load.
0ulti+user measurements are collected at % minute intervals and the focus is on
resource utilisation. #cripts used for 0ulti+,ser measurements should be
identical to scripts used for single user measurements.
If possible, at least two multi+user measurements should be taken. The first should
include all scripted transactions in equal transaction rate proportions
and the second should run transactions in the transaction mi$ e$pected to be
encountered in production. This will enable the analyst to determine the
relative impact of transactions on each other.
Post+0easurement 'ctivities
3ollowing the conclusion of the measurement e$ercise, all collected data and data
files should be delivered as soon as possible to the m5race Parsing /ngineer.
3ollowing delivery, measurement collectors are requested not to delete measurement
data located on the respective application server platforms unless
specifically requested by the m5race Performance 'nalyst.
#ubmission of 1og files
1og 3ormat
In order to determine which and how many resources were used by a specific
transaction, the start and stop times of all transactions are
required. The time periods between start and stop times are called 6end+to+end7
durations of transactions. 'll 6end+to+end7 elements are
included in the 6.ete7 file, which can be created manually or generated automatically
with a script.
In order to e$pedite the parsing process it is important to ensure that the collected
measurement information is delivered in a standard
pre+determined format. The format fields are defined described the underlying table.
3ield 3ield Data Description
DateCTime DyyyyE+DmmE+DddE DhhE(DmmE(DssE.DmsecE !#tart time can be denoted up to
si$ decimal places after the comma"
Duration In seconds.milliseconds !ma$imum of si$ decimal places after the comma"
Transaction Type ID Transaction Type ID
Transaction Description Description of Transaction ID
TimestampFT'5GDurationFT'5GTransaction ID FT'5GTransaction Description
#ingle ,ser 0easurement /$amples(
/$ample 6.ete7 file
<>>H+>H+A% %I(AH(<I.%<AB?= >.%I=>%I % #tart Portal
<>>H+>H+A% %I(AH(B<.BB?A< %.BBA< < 1ogin Portal
.erification and #ubmission of .ete+file
.erify that the transaction start and stop times are consistent with the entries in 3ield%.
#tart and stop times can be established by
converting the ,ni$ timestamp to a Date and Time on the following web site(
http(CCwww.onlineconversion.comCuni$Jtime.htm
The parsing of the measurement data will only be accepted, if the 6.ete7 file is
forwarded to the following web site( https(CCete.contentional.eu
#ystem 0onitors
The transaction profiles are established by the server resource metrics and transaction
response times obtained in the measurement
environment, namely the test environment. The following paragraphs contain
descriptions of collection methods which can be used
to collect server resource utilisation metrics !*P, usage, disk IC&, memory, etc." The
production environment is often shared by
multiple applications. It is therefore important to measure the load of the production
environment over a long time period of at
least a few months, in time intervals of about ? minutes. The file format in which log
files should be presented to the parsing
engineer is described below( KKKK00DD+DapplicationE+DmeasurementE+Dmachine
nameLMME+Dsu muLMM boLMME.Dmeasuring+toolE
The following measurement tools are readily available(
) mon
) Perfmon
) #ar
) Iostat
) Ps
) sbus
'fter each measurement, the collected measurement data should be compressed into a
.8ip file and sent to the m5race Performance 'nalyst
as soon as possible. 3ollowing delivery, measurement collectors are requested not to
delete measurement data located on the respective
application server platforms unless specifically requested by the m5race Performance
'nalyst.
Test 'utomation or #cripting for Performance 0odelling
1oadrunner
5efore 1oad;unner measurement collection can be made, any time differences
between application servers and client P*s must be established, including the load
generator P*s.
If possible, the time settings on the load generator P*s, in fact all machines and
servers should be synchronised. 't the very least, the difference in time between the
various servers should be recorded. During the measurement collection, measurement
scenarios are performed.
' measurement scenario may consisit of one or more measurement scripts. '
measurement script consists of transactions which need to be e$ecuted during the
measurement session.
The transactions are inserted into the script in the order prescribed by the business
process. /ach script is directly related to a business process or use case.

#ingle ,ser 0easurements
The measurement scenario consists of a range of use case related scripts. There is a
script for each use case. The transaction types of the use case are e$ecuted one at a
time and the use case is then repeated in five additional iterations, for a total of si$
iterations. The transaction types contained within each script are e$ecuted with ?
second fi$ed user think time intervals. The fi$ed length user think time intervals
between iterations is %> seconds. o random user think times should be used. The
transactions contained within the script are e$ecuted using a single user. #tacked
single user measurements can be e$ecuted using a number of different .users.
The 1oadrunner scenarios should be tested and checked for proper operation prior to
implementation. 'n e$clusively available test environment is not required for script
verification. The 1oadrunner scripts should be made available to the measurement
engineer at least two days prior to the measurement session. It is possible to make an
e$ception, if for e$ample 1oadrunner can only function correctly if each script can
only contain a single transaction type. In such a case, the single transaction scripts are
e$ecuted in the order determined by the use cases. The original script therefore
becomes a group of one+transaction scripts.
Immediately following the single user measurement, the 1oadrunner engineer should
perform the following actions(
) *heck the correct duration of the measurement session and report these details via
email to the performance analyst
) 4ith the help of 1oad;unner 'naly8er, create a standard report in 4ord format
) Gather a set of raw data !the 1oad;unner log" and deliver this to the performance
analyst !via email or on a shared drive"
) The #tandard report should contain the content of the e$ample report as shown in
'ppendi$ % of the manual
3or details on the log format, please see Data Gathering + #ubmission.

0ulti ,ser 0easurements
' pilot measurement e$ercise is usually conducted prior to the actual multi user
measurement session. This is preferably done at least one day prior to the actual
measurement session. &n the basis of the pilot measurement session, it is possible to
determine the limitations of transaction volumes on the test environment before the
test infrastructure becomes overburdened due to capacity limitations. The multi user
measurement scenario is conducted using a 6stepped7 approach where a number of
.users are used to e$ecute scripts for %> minutes before increasing the number of
users to the ne$t step. The stepped increase in vusers occurs every %> minutes until
the ma$imum user target is reached, as was established in the pilot measurement
session.
Transaction user think times are identical to those encounteredCused during the #ingle
,ser 0easurement session. 3ollowing the completion of the 0ulti+,ser scenario, a
standard 1oadrunner report should be created using the 1oadrunner 'nalyser utility
and the 1oadrunner raw data logs should be delivered to the performance analyst, in a
manner similar to that encountered in the single user measurement scenarios
described above.
The 1oadrunner sections which include ;unning .users, Transaction ;ate and
Transaction #ummary are of particular importance.

1ogging in 1oadrunner
This section lists a number of files which are of interest to the m5race Performance
'nalyst.
The e$ample template file 6<>>H>I<HJm5raceJ1oadrunnerJTemplate.usr7 will be
used for demonstration purposes.
The 6<>>H>I<HJm5raceJ1oadrunnerJTemplate.usr7 file contains a number of actions
that can be reused. Details of these script actions are briefly e$plained below(
vuserJinit
The vuserJinit action contains declarations of the necessary variables, the .T# dll
Mvtclient.dll7 is loaded and the connections are made with the two .T# instances.
&ne .T# instance is used to co+ordinate the #tacked #ingle ,ser measurements !port
IIII" and the other instance is used for logging purposes !port NNNN". If the port
numbers mentioned above are unavailable, other ports may be used.
globals.h
Includes the .T# header file(
:include Mvts<.hM
TestJInit
*ontains a number of required test parameters. These parameters are provided by the
*ontroller to the load generators e$ecuting the scripts and include the following(
M'pplicationM( ame of applicationO
M0easurement.ersionM( .ersion number or name of measurement, for eg.
6<>>H>I>HJProd7
M0easurementTypeM( #,, 5& of 0,
M 0easurementType.ersion M( .ersion of the measurement type, for eg. 65&+><7
The values can be passed from the *ontroller using command line parameters(
+ #elect the relevant group in the *ontroller,
+ select Details
+ select 0ore
+ 3ill in the following e$ample command line parameters(
6+'pplication P5P +0easurement.ersion <>>H>I>HJProd +0easurementType 5&
+0easurementType.ersion 5&+><7
&n the basis of the provided 0easurementType information, the value of a number of
variables are determined. The value of the 0easurementInterval variable determines
the wait time between the sub+measurements of a stacked single user measurement.
This should only be specified if the application encounters difficulties with stacked
single user scenarios that do not have wait times between stacked transactions. The
TT value is the think time used for the various types of tests used. The measurement
engineer should enter the correct think time value for the test.
5( If the script is run from .uGen, the lrJgetJattribJstring will not work correctly,
and variables can be entered instead using the lrJsaveJstring functions.
5( This action occurs after vuserJinit in the Init section. In addition, ensure that the
6Initiali8e all vuser before starting7 option has been selected on the *ontroller.
.T#
The .T# action contains three functions. /ach function is described below(
.T#JInit!" +This function checks whether the .T# server is ready at the start of the
script, and checks whether a token is available.
.T#J#tart!" +This function is used to check for the availability of a .T# token. If so,
the token is removed and two time stamps are recorded, one period time stamp and
one readable time stamp. The user then continues with the transaction.
.T#JGereed!charQ Transaction" +This function collects all relevant data during the
measurement session, formats the data to a specified format and makes an entry to the
.T# server.
The functions .T#J#tart!" and .T#JGereed!charQ Transaction" are called from the
relevant transaction script. In the template, the action function is called ,secase.
,secase
In order to service the scripted transactions, a number of functions are called. 3irst,
the usual lrJstartJtransaction and the
lrJendJtransaction. The transaction name is composed of A components(
F0easurementIdGORIterationSOF0easurementDescriptionG, whereas RIterationS is a
1oadrunner parameter of type iteration number.
The lrJstartJtransaction and the lrJendJtransaction functions are surrounded by
.T#J#tart!" and .T#JGereed
!lrJevalJstring!MF0easurementIdGORIterationSOF0easurementDescriptionGM"".
4ith the rule
;esponse time L lrJgetJtransactionJduration!lrJevalJstring!MF0easurementIdGO
RIterationSOF0easurementDescriptionGM""O
the transaction response time is written to a variable which is used for logging
purposes.
'fter every .T#JGereed function call, there is an
lrJrende8vous!MFrende8vouspointameGM" function call. ote that each rende8vous
point requires a unique name.
ote( If a multi+user measurement is being performed, rende8vous points in the script
should be disabled at the *ontroller
via #cenario @ ;ende8vouspointsT

vuserJend
In this action, the connections to the two .T# instances are broken !#ee known issues
below"
,sing .T#
5efore a measurement scenario is e$ecuted, both .T# instances should be started.
The instances can be started through the .T# G,I
!vtconsole.e$e" on the *ontroller machine, or another machine. &ne instance serves to
co+ordinate the #tacked #ingle ,ser !5&"
measurements !port IIII" and the other instance serves to provide a metrics logging
capability !port NNNN". /nter the respective
port number in the 6.T# Port 'ddress7 option bo$ and select 6#tart7. The correct
column names can subsequently be imported via
&ptions @Import data option. The e$isting .T#J5&.t$t file is used for the 5& co+
ordination instance, and the .T#J1ogging.t$t
file is used for the .T# instance dedicated to the logging requirement.
4hen a measurement has been completed, the .T# results can be e$ported to a 6.csv7
file using the &ptions @ /$port data functionality.

Unown Issues
) 't the end of a 5& measurement session a runtime error appears if the vusers
attempt to terminate their connections to the .T#
server simultaneously. The root cause of this runtime error is still under investigation.
The runtime error does not appear to be
a problem, as the .T# G,I -ust keeps running and the log data can be e$ported
without any issues.
) If one of the sub+measurements of a 5& measurement encounters a fault during
e$ecution, the .T# co+ordination mechanism no longer
functions correctly as the token is not returned to .T# server instance for the ne$t
.user. This can be prevented by identifying possible
failure points in the script, capturing fault events, and returning the token to the .T#
instance using the .T#JGereed function.
In case of such failure events with a 5& measurement, the analyst should be
wondering what the value of a 5& measurement is if it
contains failed transactions.
0easurement data should be 68ipped7 up and delivered to the m5race Performance
'nalyst as soon as they become available. 'ny measurement
data should not be deleted, unless the m5race Performance 'nalyst has indicated that
the measurements were safely received and that it is
safe to delete the data. It would be prudent to ensure that at least two copies of the
measurement data e$ist at any one time.
Presentation of 0etrics for Parsing
*ollected metric data should be uploaded to the 3TP site( ftp(CCftp.contentional.eu in
the following manner(
The file and directory structure can be constructed from the data elements in the 6.ete7
file, see V <.
WD;elationshipEWD3ield?EWD3ield=EWD3ield%%E
D;elationshipE L ame of relationshipCcustomer
D3ield?E L Pro-ect Identifier
D3ield=E L 0easurement umber or Phase umber
D3ield%%E L #,, 5& or 0, sequence number
The following is an e$ample of the directory and file structure used for the storage of
a number of measurements for a fictional pro-ect
called P5P at a customer called XKP Insurance(
WXKPJInsurance
Y++ P5P
Y++ Phase<
Y++ 5&
Y Y++ <>>H%>%%+P5P+Phase<+5&.ete
Y Y++ <>>H%>%%+P5P+Phase<+servJ>>=B+5&.perfmon
Y Y++ <>>H%>%%+P5P+Phase<+apsapu<=+5&.nmon
Y++ 0,%
Y Y++ <>>H%>%%+P5P+Phase<+0,%.ete
Y Y++ <>>H%>%%+P5P+Phase<+servJ>>=B+0,%.perfmon
Y Y++ <>>H%>%%+P5P+Phase<+apsapu<=+0,%.nmon
Y++ 0,<
Y Y++ <>>H%>%%+P5P+Phase<+0,<.ete
Y Y++ <>>H%>%%+P5P+Phase<+servJ>>=B+0,<.perfmon
Y Y++ <>>H%>%%+P5P+Phase<+apsapu<=+0,<.nmon
Y++ #,
Y++ <>>H%>%%+P5P+Phase<+#,.ete
Y++ <>>H%>%%+P5P+Phase<+servJ>>=B+#,.perfmon
Y++ <>>H%>%%+P5P+Phase<+apsapu<=+#,.nmon
ote that dates are in KK00DD format, and that any spaces in names should be
replaced with an underscore !6J7".