Lec#3 - Selection of Techniques and Metrics PDF

Simulation, Modeling and Analysis of
Computer Networks
(ECE 6620)
Selection of Techniques and Metrics

(Chapter#3)
Art of Computer Systems Performance Analysis

By R. Jain
Dr. M. Hasan Islam
2010 Raj Jain www.rajjain.com
3-1
Overview
Criteria for Selecting an Evaluation Technique
Three Rules of Validation
Selecting Performance Metrics
Commonly Used Performance Metrics
Utility Classification of Metrics
Setting Performance Requirements
3-2
Criteria for Selecting an Evaluation Technique
3-3
Three Rules of Validation
Do not trust the results of a simulation model until

they have been validated by analytical modeling or
measurements.
Do not trust the results of an analytical model until

they have been validated by a simulation model or
measurements.
Do not trust the results of a measurement until they

have been validated by simulation or analytical
modeling.
3-4
Selecting Performance Metrics
3-5
Selecting Metrics
Include:
Performance Time, Rate, Resource
Error rate, probability
Time to failure and duration
Consider including:
Mean and variance
Individual and Global
Selection Criteria:
Low-variability
Non-redundancy
Completeness
3-6
Case Study: Two Congestion Control Algorithms
Compare two different congestion control algorithms

for computer networks
A computer network consists of a number of end systems
interconnected via a number of intermediate systems
End systems send packets to other end systems on the

network
Intermediate systems forward packets along the right path
The problem of congestion occurs when the number of

packets waiting at an intermediate system exceeds the
systems buffering capacity and some of the packets have
to be dropped
3-7
Case Study: Two Congestion Control Algorithms

System: Consists of networks
Service: Send packets from specified source to
specified destination in order
Possible outcomes:
Some packets are delivered in order to the correct

destination
Some packets are delivered out-of-order to the destination
Some packets are delivered more than once (duplicates)
Some packets are dropped on the way (lost packets)
3-8
Performance metrics: For packets delivered in order
Time-rate-resource metrics
Response time to deliver the packets
Throughput
Delay inside the network for individual packets

Determines the time that a packet has to be kept at the source end
station using up its memory resources
Lower response time is considered better
Number of packets per unit of time
Larger throughput is considered better
Processor time per packet on the source end system

Processor time per packet on the destination end systems
Processor time per packet on the intermediate systems
3-9
Variability of the response time
Probability of out-of-order arrivals
Highly variant response results in unnecessary retransmissions

Undesirable since they cannot generally be delivered to the user
immediately
In many systems, the out-of-order packets are discarded at the
destination end systems
In others, they are stored in system buffers awaiting arrival of
intervening packets
In either case, out-of-order arrivals cause additional overhead
(buffers)
Probability of duplicate packets
Consume the network resources without any use

3-10
Probability of Lost packets
Undesirable results in retransmissions, delays, loss of data,

disconnection
Probably of disconnect
Excessive losses result in excessive retransmissions and could cause

some user connections to be broken prematurely
3-11
Case Study (Cont)
Fairness
The network is a multiuser system. It is necessary that all users be

treated fairly. It is defined as a function of variability of
throughput across users
For any given set of user throughputs (x1, x2,..., xn), the following
function can be used to assign a fairness index to the set:
Fairness Index Properties:

For all non- negative values it is always lies between 0 and 1
If all users receive equal throughput, the fairness index is 1
If k of n receive x and n-k users receive zero throughput: the fairness
index is k/n.
3-12
Redundant Parameters
After a few experiments, throughput and delay were found

redundant
All schemes that resulted in higher throughput also resulted in
higher delay, therefore, the two metrics were removed from the
list and instead a combined metric called power
A higher power meant either a higher throughput or a lower

delay
3-13
Redundant Parameters
Variance in response time

Also dropped since it was redundant with the probability of
duplication and the probability of disconnection
A higher variance resulted in a higher probability of
duplication and a higher probability of premature
disconnection
Thus, in this study a set of nine metrics were used to

compare different congestion control algorithms
3-14
Response time is defined as the interval between a

users request and the system response
Simplistic definition
3-15
Two definitions of response time
3-16
Turnaround time
Basically the responsiveness for a batch stream

Time between the submission of a batch job and the
completion of its output
Time to read the input is included in the turnaround time
Reaction Time
Time between submission of a request and the beginning of

its execution by the system
To measure the reaction time, one has to able to monitor the
actions inside a system since the beginning of the execution
may not correspond to any externally visible event
3-17
Stretch factor
Ratio of response time at a particular load to that at
the minimum load
Response time of a system generally increases as
the load on the system increases
For a timesharing system, for example, the stretch
factor is defined as the ratio of the response time
with multiprogramming to that without
multiprogramming
3-18
Common Performance Metrics (cont)
Throughput
Rate (requests per unit of time) at which the requests can be

serviced by the system
For batch streams, the throughput is measured in jobs per second
For interactive systems, throughput is measured in requests per
second
Jobs per second

Requests per second
Millions of Instructions Per Second (MIPS)
Millions of Floating Point Operations Per Second (MFLOPS)
Packets Per Second (PPS)
Bits per second (bps)
Transactions Per Second (TPS)
3-19
Common Performance Metrics (Cont)
Throughput of a system generally increases as the load on the system

initially increases and after a certain load, the throughput stops increasing;
in most cases, it may even start decreasing
Nominal Capacity: Maximum achievable throughput under ideal workload

conditions. e.g., for networks nominal capacity is called bandwidth in bits
per second. The response time at maximum throughput is too high
Usable capacity: Maximum throughput achievable without exceeding a

pre-specified response-time limit
Knee Capacity: throughput at Knee = Low response time and High

throughput
It is also common to measure capacity in terms of load, for example, the

number of users rather than the throughput
3-20
Capacity
3-21
Efficiency: Ratio of maximum usable capacity (achievable throughput) to

nominal capacity
Maximum throughput from a 100-Mbps (megabits per second) Local
Area Network (LAN) is only 85 Mbps, its efficiency is 85%.
For multi-processors system ratio of the performance of an n-processor
system to that of a one-processor system is its efficiency
Utilization: The fraction of time the resource is busy servicing requests.
Average fraction used for memory
processors, are always either busy or
idle, so their utilization is in terms of
ratio of busy time to total time. For other
resources, such as memory, only a
fraction of the resource may be
used at a given time; their utilization is
measured as the average fraction used
over an interval
3-22
Reliability:
Usually measured by probability of errors or the mean time between errors

(error-free seconds)
Availability:
Fraction of the time the system is available to service users requests
Time during which the system is not available is called downtime; the
time during which the system is available is called uptime
Better indicator of uptime Mean Time to Failure (MTTF)
Mean Time to Repair (MTTR)
MTTF/(MTTF+MTTR)
3-23

In system procurement studies, the cost/performance
ratio is commonly used as a metric for comparing two
or more systems
The cost includes the cost of hardware/software
licensing, installation, and maintenance over a given
number of years
The performance is measured in terms of throughput
under a given response time constraint
For example, two transaction processing systems may
be compared in terms of dollars per TPS
3-24
Utility function of a performance metric, can be categorized into three classes

Higher is Better or HB
System users and system managers prefer higher values of such metrics.
System throughput is an example of an HB metric.
Lower is Better or LB
System users and system managers prefer smaller values of such metrics.
Response time is an example of an LB metric
Nominal is Best or NB
Both high and low values are undesirable. A particular value in the middle is
considered the best
Very high utilization is considered bad by the users since their response times
are high
Very low utilization is considered bad by system managers since the system
resources are not being used
Some value in the range of 50 to 75% may be considered best by both users
and system managers
3-25
3-26

One problem performance analysts are faced with is
that of specifying performance requirements for a
system to be acquired or designed
Typical system requirements - Examples:
The system should be both processing and memory
efficient. It should not create excessive overhead
There should be an extremely low probability that
the network will duplicate a packet, deliver a
packet to the wrong destination, or change the data
in a packet.
3-27
General Problems (may not applied to all):

Non-Specific - No clear numbers are specified. Qualitative words such as
low, high, rare, and extremely small are used instead
Non-Measurable - no way to measure a system and verify that it meets the
requirement
Non-Acceptable - Numerical values of requirements, if specified, are set
based upon what can be achieved or what looks good. If an attempt is
made to set the requirements realistically, they turn out to be so low that
they become unacceptable (achievable)
Non-Realizable - Often, requirements are set high so that they look good.
However, such requirements may not be realizable (achievable)
Non-Thorough - No attempt is made to specify a possible outcomes
(study of all possible outcomes and failures)
SMART
Specific, Measurable, Acceptable, Realizable, and Thorough
3-28
Case Study 3.2: Local Area Networks

A LAN basically provides the service of transporting
frames (or packets) to the specified destination station
Given a user request to send a frame to destination
station D, there are three categories of outcomes:
Frame is correctly delivered to D
Incorrectly delivered (delivered to a wrong
destination or with an error indication to D)
Not delivered at all
Service: Send frame to Destnation

3-29
Performance requirements for three categories

Speed: If the packet is correctly delivered, the time
taken to deliver it and the rate at which it is
delivered are important. This leads to the following
two requirements:
Access delay at any station should be less than 1 second
Sustained throughput must be at least 80 Mbits/sec
3-30
Reliability: Six different error modes were considered important.

Probability requirements for each of these error modes and their
combined effect are specified as follows:
of any bit being in error must be less than 10-7
Probability of any frame being in error (with error indication set)
must be less than 1%
Probability of a frame in error being delivered without error
indication must be less than 10-15
Probability of a frame being mis-delivered due to an undetected
error in the destination address must be less than 10-18
Probability of a frame being delivered more than once (duplicate)
must be less than 10-5
Probability of losing a frame (due to all sorts of errors) must be less
than1%
Probability
3-31
Availability: Two fault modes were considered significant

First - time lost due to the network re-initializations
Second - Time lost due to permanent failures requiring field
service calls
Requirements for frequency and duration of these faults are:
Mean time to initialize LAN must be less than 15 milliseconds

Mean time between LAN initializations must be at least 1 minute
Mean time to repair a LAN must be less than 1 hour
Mean time between LAN partitioning must be at least half a week
All of the numerical values specified above were checked for realizability by
analytical modeling, which showed that LAN systems satisfying these
requirements were feasible
3-32
Assignment-2
Make a complete list of metrics to compare

Two personal computers
Two database systems
Two disk drives
Two window systems
Submission Date: 17 Feb 12pm
3-33

Lec#3 - Selection of Techniques and Metrics PDF

Hochgeladen von

Dokumentinformationen

Originalbeschreibung:

Originaltitel

Copyright

Verfügbare Formate

Dieses Dokument teilen

Dokument teilen oder einbetten

Freigabeoptionen

Stufen Sie dieses Dokument als nützlich ein?

Sind diese Inhalte unangemessen?

Copyright:

Verfügbare Formate

Lec#3 - Selection of Techniques and Metrics PDF

Hochgeladen von

Copyright:

Verfügbare Formate

Simulation, Modeling and Analysis of

Selection of Techniques and Metrics

Art of Computer Systems Performance Analysis

Dr. M. Hasan Islam

2010 Raj Jain www.rajjain.com

2010 Raj Jain www.rajjain.com

Criteria for Selecting an Evaluation Technique

2010 Raj Jain www.rajjain.com

Three Rules of Validation

Do not trust the results of a simulation model until

Do not trust the results of an analytical model until

Do not trust the results of a measurement until they

Selecting Performance Metrics

2010 Raj Jain www.rajjain.com

2010 Raj Jain www.rajjain.com

Case Study: Two Congestion Control Algorithms

Compare two different congestion control algorithms

End systems send packets to other end systems on the

The problem of congestion occurs when the number of

Case Study: Two Congestion Control Algorithms

Some packets are delivered in order to the correct

2010 Raj Jain www.rajjain.com

Performance metrics: For packets delivered in order

Delay inside the network for individual packets

Processor time per packet on the source end system

Performance metrics: For packets delivered in order

Variability of the response time

Probability of out-of-order arrivals

Highly variant response results in unnecessary retransmissions

Probability of duplicate packets

Consume the network resources without any use

Performance metrics: For packets delivered in order

Probability of Lost packets

Undesirable results in retransmissions, delays, loss of data,

Excessive losses result in excessive retransmissions and could cause

2010 Raj Jain www.rajjain.com

Case Study (Cont)

The network is a multiuser system. It is necessary that all users be

Fairness Index Properties:

After a few experiments, throughput and delay were found

A higher power meant either a higher throughput or a lower

2010 Raj Jain www.rajjain.com

Variance in response time

Thus, in this study a set of nine metrics were used to

2010 Raj Jain www.rajjain.com

Commonly Used Performance Metrics

Response time is defined as the interval between a

2010 Raj Jain www.rajjain.com

Commonly Used Performance Metrics

Two definitions of response time

2010 Raj Jain www.rajjain.com

Commonly Used Performance Metrics

Basically the responsiveness for a batch stream

Time between submission of a request and the beginning of

Commonly Used Performance Metrics

2010 Raj Jain www.rajjain.com

Common Performance Metrics (cont)

Rate (requests per unit of time) at which the requests can be

Jobs per second

Common Performance Metrics (Cont)

Throughput of a system generally increases as the load on the system

Nominal Capacity: Maximum achievable throughput under ideal workload