AppTransaction Xpert

Lecture Manual
Introduction to ACE Analyst
NOTICES
Product Information
Lecture Manual for: ACE Analyst
Software Release: 16.0.T PL0
Destination Control Statement

This Lab Manual and the OPNET Software were
manufactured in the United States by OPNET. The initial
export of this Lab Manual and the OPNET Software, and
any subsequent relocation to another country shall comply
with United States law relating to the export of technical
data, equipment, software and know-how.
Technical Support
Restricted Rights Legend

Telephone:
240.497.1200
Facsimile:
240.497.1064
E-mail:
support@opnet.com
Web:
http://www.opnet.com
OPNET Technologies, Inc. (OPNET) is the sole owner

of all proprietary rights in this Lab Manual (the Lab
Manual) and the OPNET IT Guru software (the
OPNET Software).
Terms and Conditions of Use
Access to and use of the Training Manual and/or the
OPNET Software is restricted to authorized users who
are employees or direct contractors of a Licensee who
has accepted the terms and conditions of OPNETs
Software Master License Agreement. As a condition of
having access to or use of the Lab Manual and/or the
OPNET Software all users agree to be subject the terms
of OPNETs Software Master License Agreement. Users
shall not copy or reproduce, in whole or in part, or
permit access to or disclose the Lab Manual or the
OPNET Software to any other person without the
prior written consent of OPNET.
Liability Statement
The terms and conditions governing the licensing of
OPNET Software consist of those set forth in the written
contracts and license agreements between OPNET and its
customers. No representations in this Lab Manual or
statements of capability, suitability for use, accuracy or
performance, shall be a warranty by OPNET unless
contained in a written agreement signed by OPNET.
OPNET DISCLAIMS ALL EXPRESS OR IMPLIED
WARRANTIES AS TO THE LAB MANUAL AND
THE OPNET SOFTWARE INCLUDING IMPLIED
WARRANTIES OF MERCHANTABILITY AND
FITNESS FOR A PARTICULAR PURPOSE.
USE BY AGENCIES OF THE UNITED STATES

GOVERNMENT: For purposes of use of this Lab
Manual and OPNET Software by agencies of the United
States government (or those authorized to use of this Lab
Manual and OPNET Software on behalf of an agency),
the OPNET Software is Commercial Computer
Software within the meaning of DFARs 227.7202 and
252.227-7014(a)(1) and FARs 52.227-19. Use,
reproduction and disclosure is subject to DFAR and FAR
provisions relating to Commercial Computer Software
and the License Agreement between OPNET and the
agencies using OPNET Software. OPNET software is
provided with a RESTRICTED RIGHTS NOTICE for
government use.
Trademarks
OPNET, OPNET Technologies, Inc., OPNET Modeler, IT
Guru, OPNET Development Kit, NetBiz and IT Guru are
trademarks of OPNET Technologies, Inc. All other
trademarks used in the Lab Manual are the property of their
respective owners and used herein for identification
purposes only.
CONFIDENTIAL INFORMATION
DO NOT DISCLOSE, FORWARD, DISTRIBUTE,
SHARE, OR MAKE COPIES OF THIS DOCUMENT IN
WHOLE OR IN PART. This document contains
confidential information and may contain information that
is proprietary, privileged, and/or exempt from disclosure
under applicable law. This document is intended for the
exclusive use of the person to whom it is disclosed. If you
are an unauthorized person, you are hereby notified that
any viewing, copying, disclosure or distribution of this
information may be subject to legal action. All
unauthorized persons must immediately destroy the original
documentation without making any copies or further
unauthorized disclosure.
Lecture Manual
Introduction to ACE Analyst
NOTICES
Product Information
Lecture Manual for: ACE Analyst
Software Release: 16.0.T PL0
Destination Control Statement

This Lab Manual and the OPNET Software were
manufactured in the United States by OPNET. The initial
export of this Lab Manual and the OPNET Software, and
any subsequent relocation to another country shall comply
with United States law relating to the export of technical
data, equipment, software and know-how.
Technical Support
Restricted Rights Legend

Telephone:
240.497.1200
Facsimile:
240.497.1064
E-mail:
support@opnet.com
Web:
http://www.opnet.com
OPNET Technologies, Inc. (OPNET) is the sole owner

of all proprietary rights in this Lab Manual (the Lab
Manual) and the OPNET IT Guru software (the
OPNET Software).
Terms and Conditions of Use
Access to and use of the Training Manual and/or the
OPNET Software is restricted to authorized users who
are employees or direct contractors of a Licensee who
has accepted the terms and conditions of OPNETs
Software Master License Agreement. As a condition of
having access to or use of the Lab Manual and/or the
OPNET Software all users agree to be subject the terms
of OPNETs Software Master License Agreement. Users
shall not copy or reproduce, in whole or in part, or
permit access to or disclose the Lab Manual or the
OPNET Software to any other person without the
prior written consent of OPNET.
Liability Statement
The terms and conditions governing the licensing of
OPNET Software consist of those set forth in the written
contracts and license agreements between OPNET and its
customers. No representations in this Lab Manual or
statements of capability, suitability for use, accuracy or
performance, shall be a warranty by OPNET unless
contained in a written agreement signed by OPNET.
OPNET DISCLAIMS ALL EXPRESS OR IMPLIED
WARRANTIES AS TO THE LAB MANUAL AND
THE OPNET SOFTWARE INCLUDING IMPLIED
WARRANTIES OF MERCHANTABILITY AND
FITNESS FOR A PARTICULAR PURPOSE.
USE BY AGENCIES OF THE UNITED STATES

GOVERNMENT: For purposes of use of this Lab
Manual and OPNET Software by agencies of the United
States government (or those authorized to use of this Lab
Manual and OPNET Software on behalf of an agency),
the OPNET Software is Commercial Computer
Software within the meaning of DFARs 227.7202 and
252.227-7014(a)(1) and FARs 52.227-19. Use,
reproduction and disclosure is subject to DFAR and FAR
provisions relating to Commercial Computer Software
and the License Agreement between OPNET and the
agencies using OPNET Software. OPNET software is
provided with a RESTRICTED RIGHTS NOTICE for
government use.
Trademarks
OPNET, OPNET Technologies, Inc., OPNET Modeler, IT
Guru, OPNET Development Kit, NetBiz and IT Guru are
trademarks of OPNET Technologies, Inc. All other
trademarks used in the Lab Manual are the property of their
respective owners and used herein for identification
purposes only.
CONFIDENTIAL INFORMATION
DO NOT DISCLOSE, FORWARD, DISTRIBUTE,
SHARE, OR MAKE COPIES OF THIS DOCUMENT IN
WHOLE OR IN PART. This document contains
confidential information and may contain information that
is proprietary, privileged, and/or exempt from disclosure
under applicable law. This document is intended for the
exclusive use of the person to whom it is disclosed. If you
are an unauthorized person, you are hereby notified that
any viewing, copying, disclosure or distribution of this
information may be subject to legal action. All
unauthorized persons must immediately destroy the original
documentation without making any copies or further
unauthorized disclosure.
2/10/2011
Introduction to AppTransaction
Xpert (ACE Analyst)
test
Day 1
OPNET Confidential Not for release to third parties 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc.
About OPNET Technologies, Inc.

Corporate Overview
Founded in 1986
Publicly traded (NASDAQ: OPNT)
HQ in Bethesda, MD
Approximately 600 employees
Worldwide presence through direct offices and channel
partners
Global distribution agreement with Cisco
Best-in-Class Solutions and Services
Application Performance Management
Network Performance Management
Network R&D
Strong Financial Track Record
g historyy of profitability
p
y
Long
Trailing 12-month revenue of over $100M
Approximatelytest
25% of revenue re-invested in R&D
Broad Customer Base
Corporate Enterprises
Government Agencies/DoD
Service Providers
Network Equipment Manufacturers
3
OPNET Confidential Not for release to third parties 2010 OPNET Technologies, Inc. All
rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies,
Inc.
2/10/2011
Customers (partial list)

Financial Services
Government
Insurance
Healthcare / Bio / Pharmaceutical
Manufacturing
Energy / Utilities
Media / Communications
Services
test
Retail / Consumer Products
Transportation / Logistics
OPNETs APM Xpert Suite

End-user experience and network
monitoring and analytics
Deep application component

monitoring and analytics
Deep transaction tracing,

analysis, and prediction
Passive database
performance monitoring
test
Network and server

discovery and monitoring
What's in your Application?
OPNET Confidential Not for release to third parties

2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc.
Inc.
2/10/2011
OPNET is an IACET Authorized Provider

OPNET Technologies, Inc. has been approved as an Authorized Provider by the
International Association for Continuing Education and Training (IACET), 1760 Old
Meadow Road, Suite 500, McLean, VA 22102. In obtaining this approval, OPNET
has demonstrated that it complies with the ANSI/IACET 11-2007
2007 Standard which is
internationally recognized as the Standard for training organizations.
As a result of their Authorized Provider status OPNET is authorized to offer IACET
Continuing Education Units (CEUs) for its programs that qualify under the
ANSI/IACET 1-2007 Standard.
test
Continuing Education Units

What is IACET?
IACET is a Standards Developing Organization accredited by the American Standards Institute
(ANSI) and is an authorizing body for education providers that meet the American National
Standard for continuing education and training (ANSI/IACET 1-2007 Standard). The IACET
standard is what learners seek for quality and is the core of thousands of educational programs
worldwide. (see: www.IACET.org)
What is an Authorized Provider?
The IACET Authorized Provider (AP) status means that an organization adheres to IACETs
internationally recognized ANSI/IACET 1-2007 Standard for Continuing Education and Training.
Only Authorized Providers may award IACET Continuing Education Units (CEUs).
Whatt is
Wh
i a CEU?
test
The Continuing Education
Unit (CEU) was created by IACET as a measurement of continuing
education. One (1) IACET CEU is equal to ten (10) contact hours of participation in an organized
continuing education experience under responsible sponsorship, capable direction, and qualified
instruction.
Inc.
2/10/2011
Earning CEUs for this Course

Introduction to AppTransaction Xpert
This course is worth 1.3 CEUs
To receive these CEUs, you must
Attend the entire course

Fill out the feedback form
Score 80% in a test at the end of class
Your test is scored in the next 10 business days

If successful, you will receive a certificate denoting your CEUs in the mail
within 2 weeks
test of the course is stored in our database for later retrieval as
Your completion
needed
Lets get to know each other

Name ?
What company you work for ?
Which Department ?
What experience do you have with OPNET products?
Did you know we changed the names !!??
test
Inc.
2/10/2011
Course Structure
10
Day 1
Introduction to AppTransaction Xpert
Day 2
Advanced AppTransaction Xpert
About OPNET
Visualizations in AppTransaction Xpert
Troubleshooting application
performance
Predicting application behavior before
deployment
test strategies
Capture and import
WAN Acceleration
Advanced Topics
Understanding Delay Components
Parallel Effects, User think Time
Troubleshooting TCP applications
Class Format
In general, four phase instruction
Short lecture sections
Demonstrations
Related hands-on labs
Discussion of lab applications
Class is interactive dont hesitate to ask questions

Please fill out the feedback form online @ eval.opnet.com
Give us information on your current network and application concerns
test
Provide suggestions on how we can improve our products
Let us know how we can improve this training class
11
Inc.
2/10/2011
Agenda
Introduction
- Licensing Options
- Overview, Use Cases
- Basic Workflow
Using AppTransaction Xpert to visualize applications

- AppTransaction Xpert Visualizations
- Lab 1: Visualizing Application
Using AppTransaction Xpert to diagnose problematic applications

- AppDoctor
Reporting capabilities of AppTransaction Xpert

- Reports
R
t
test
- Lab 2: Application
Performance Troubleshooting
Using AppTransaction Xpert to perform performance predictive studies

- QuickPredict and QuickRecode
- Lab 3: Predictive Analysis
12
Licensing Options
Download and administer licenses using the License Manager
Licensed on concurrent user basis
Online license management
g
(http://www.opnet.com/support/license_management.html)
Licensing options
Standalone vs. Floating (license server)
Loanable Licenses
test
13
Inc.
2/10/2011
Overview of AppTransaction
Xpert
test
AppTransaction Xpert Solution

AppTransaction Xpert Plus
AppTransaction Xpert Standard
IT Guru with the following modules
ACE
ACE Decode Module
Advanced ACE Capabilities
Starting with version 16.0, the different offerings of AppTransaction Xpert (that is:
AppTransaction Xpert Plus, AppTransaction Xpert Standard, and AppTransaction
Xpert Module) have been combined into a single installer.
test
ImportantExisting users are not losing any AppTransaction Xpert functionality

because of this change. Functionality is determined by licenses.
15
Inc.
2/10/2011
What is AppTransaction Xpert... ?

Application Characterization Environment
Capture application traces
Visualize
Vi li Application
A li ti Dynamics
D
i
Application layer analysis
Network layer analysis
Diagnose performance problems

Minimize troubleshooting time
Validate solutions
test
Predict application behavior under different network conditions

Evaluate performance impact of application design decisions
Validate infrastructure readiness for new deployments
Generate reports
16
When to use AppTransaction

Xpert ??
(Use Cases)
test
Inc.
2/10/2011
Use Case 1
Troubleshooting application performance
Users of a critical application are reporting very bad response times

Network engineers are reporting no issues on the network
Application engineers are reporting no issues with software
Result: Finger pointing !!

Wasted time
test
18
Use Case 2
Testing changes of the enterprise environment
Relocation of key processing data center from US to China

Management needs to know the implications of move
Impact on application performance and network utilization?
Will additional resources be required?
Result: Need to produce accurate performance projections and network

planning
l i requirements
i
test
19
Inc.
2/10/2011
Use Case 3
Application pre-deployment analysis
Purchase of a new 3-party application system

Management needs to know how the new system will perform
Application response times
Network utilization
Result: Need to eliminate the need for trial and error deployment of
application. Expensive and time consuming.
test
20
Basic Workflow
test
Inc.
10
2/10/2011
What does AppTransaction Xpert tell you ?

Why is my application slow?
Users desktop running slow?

Server delays?
y
Non-optimal protocol settings (TCP and others)?
Heavy congestion over WAN?
Too little bandwidth?
Application too chatty?
Can this new application be deployed over our network?

Expected test
transaction response times?
Impact other critical applications?
Bandwidth requirements?
22
How does AppTransaction Xpert tell you this...?

Reports, graphs, and tables
Visualization of application
pp
behavior
Diagnosis of bottlenecks and performance problems
Predictive analysis of same traffic on different network
Input to other OPNET products for further analysis
Application traffic analysis
test
Capacity planning studies
23
Inc.
11
2/10/2011
Inputs to AppTransaction Xpert

Packet traces from the following sources
ACE Capture Agents
ACE Live
Cisco NAM Blade
WAN Accelerator
F5 BIG-IP
Sniffer
UNIX (no agent installed)
Tcpdump, windump ..etc
Basic characteristics of network (Recommended)

test
Tier locations
Bandwidth between tiers
Latency between tiers
24
Application Troubleshooting Workflow

Identify performance issues
Capture Packet Traces
Visualize, Diagnose, and Predict
Optional: Detailed
Analysis
test
Optional: Create IT Guru

network model
Generate Reports
25
Inc.
12
2/10/2011
AppTransaction Xpert
Visualizations
test
AppTransaction Xpert Visualizations

Data Exchange Chart
Shows timeline of data sent
Tier Pair Circle

Shows summary statistics
Tree View
Shows Transaction Analysis
Decodes
Shows Application layer headers
Graphs
test
View time-based graphs of application statistics
27
Inc.
13
2/10/2011
Data Exchange Chart (DEC)

Timeline of application
messages and network
packets between tiers
Slope indicates time to cross

network
Color indicates payload size

of message
test
28
DEC: Message Groups

When messages and packet
are too frequent to be displayed
individually, a solid block is shown
Block is a histogram showing the
size of messages in groups
Zoom in or use tooltip to see more
detail
test
29
Inc.
14
2/10/2011
DEC: View Options

Application Chart Only
Shows every application message several packets may comprise a message
Network Chart Only
Shows every individual packet in the trace file, including TCP acknowledgements
Application and Network Charts
Shows both charts in the same window
test
Application Messages
Network Packets
30
DEC: Show Dependencies

Triangles appear at largest
chunks of delay
Click Show Dependencies
White = delay at host

Orange = user think time
Purple = network delay
test
Legend
31
Inc.
15
2/10/2011
DEC: Application Characteristics

Quiet Time
User think time?

Slow client computer?
p
Slow server?
Not seeing the whole picture?
Tip: See what happened before and

f silence
after
test
Tip: Check Network Chart for

more details
Large periods of silence
32
DEC: Application Characteristics (cont.)

Chatty Application
Lots of back-and-forth communication
(Application Turns)
Lots of small messages

(orange and yellow)
test
Groups of messages close together
33
Inc.
16
2/10/2011
Application Turn
Counted on the Application Layer (Application Messages)
Every turn has to cross the network latency which adds up to the total
Latency Delay
2
Client
Network
Latency
test
Server
1
34
DEC: Application Characteristics

One-way data transfers
Dark blue arrows
Most arrows in same direction
Tip: Check the Network Packet

Chart for more details
test
Zoom to see the arrow heads
35
Inc.
17
2/10/2011
DEC: Application Characteristics (cont.)

Lots of irrelevant
traffic
Do not try diagnose
application until
you removed
irrelevant data
test
Tip: Go to Tier Pair

Circle and delete
unrelated traffic
36
Tier Pair Circle

Summary of traffic between all tiers
g
A high-level
view of the
application task
Is there extraneous traffic?
Right-click to exclude
unwanted traffic
test
Tip: If you dont see the protocol you expect, then right-click to re-recode traffic on the
fly
37
Inc.
18
2/10/2011
Tier Pair Circle: Functionality

Choose statistics to view
Right-click a tier or
conversation to exclude
/delete it or all others
Show only traffic containing

a specific protocol
Find Top
p Statistics to sort
conversations by
relevance
test
38
Tier Pair Circle: Statistics

Pull down to select statistics to
view:
test
39
Inc.
19
2/10/2011
Tree View
Tree pane: One-line summary of tier pairs, connections and frames
Table pane: Detail statistics by tier pairs, connections and frames
Timeline ppane: Overview of messages
g over the time of the task
test
Tree Pane
40
Table Pane
Timeline Pane
Tree View: View by: Options

Tier Pairs Network Packets
Tier Pairs Application Transactions
Application Transactions
Group by:
test
41
Inc.
20
2/10/2011
Tree View: Sorting

Click a column header to sort the table by descending values.
Click the table header again to sort by ascending values.
test
42
Tree View : Customize the Table Pane

Choose View > Choose Table Columns
Right-click on the column header and click Choose TreeView
Columns
Select the network statistics to appear in the table
test
43
Inc.
21
2/10/2011
Decode Viewer
ETH
IP
TCP
HTTP
ETH
What are decodes?

Analysis or description of the bytes at each layer of a packet
What can you test

learn from decodes?
Detailed protocol information
Information about data being transferred
Ex: TCP gives Sequence number and Acknowledgement number
44
Embedded Decode Viewer

Show / Hide the
embedded decode viewer
by clicking
and
Select traffic in the top

pane and see
corresponding decodes in
the bottom pane
Right-click on selected
frames to export to
test
spreadsheet
Click the plus sign to

expand layers of decodes
45
Inc.
22
2/10/2011
Tree View to DEC

Display Context-Based Traffic in Data Exchange Chart
Double-click on traffic for a tier pair in the Timeline pane to display the
corresponding context- based traffic in the Data Exchange Chart.
test
46
Graphs
Time varying statistics
Advanced diagnosis of network
and TCP problems
test
47
Inc.
23
2/10/2011
Graphs (cont.)
Select graphs to display
Options:
Specify bucket width for
precision of graph
Embed graphs in DEC for
easy comparison against
messages in DEC
test
Overlay similar
graphs or
overlay all
Select Show
48
Graphs: Bucket Width

What is a bucket width?
Statistics can be averaged over
time rather than shown as
individual data points
A bucket is the duration of data
you want to average, like 100 ms
or 3 seconds
Same trace, same statistics,

d e e t bucket
different
buc et widths:
w dt s:
100 ms
test
3000 ms (3 seconds)
49
Inc.
24
2/10/2011
Graphs: Embedded Statistics

Time-aligned statistics with the Data Exchange Chart
test
50
LAB 1
LAB 1
Visualizing
Application
test
51
Inc.
25
2/10/2011
Lab 1: Visualizing Application

Overview
The ACE trace file represents
p
a three-tier Oracle application.
pp
You will
explore some of the different visualizations in AppTransaction Xpert that
you can use to troubleshoot and analyze the application.
Starts on page 1 in your lab manual
test
52
Lab 1: Conclusion
AppTransaction Xpert provides many different views of a transaction
to understand the interaction of the application on the network
Data Exchange Chart

Shows timeline of data sent
Tier Pair Circle

Shows who talks to whom and summary statistics
Tree View
View the network packets in order, with a decode summary
Transaction analysis
See the timeline of transactions
Decodes
Drill downtest
into TCP, IP, or even application layer headers to understand
application behavior
Graphs
View time-based graphs of throughput, congestion, and other statistics
53
Inc.
26
2/10/2011
AppDoctor
test
What is AppDoctor ?
Invoke AppDoctor
AppDoctor (menu)
AppDoctor (icon)
AppDoctor Tabs
Summary of Delays
Executive Summary
Diagnosis
Statistics
Protocol-specific
test
55
Inc.
27
2/10/2011
AppDoctor: Summary of Delays

Pie chart that breaks down the
total response time into
processing, network, and
parallel
ll l effects
ff t
Most useful view in

AppTransaction Xpert when
troubleshooting applications
Quickly points you to what is

the problem, or at least what is
test
definitely not the problem
Must be used on a single

application task
56
Summary of Delays: Processing Effects

Tier Processing delay
Time spent processing on each tier
test
57
Inc.
28
2/10/2011
Summary of Delays: Network Effects

Latency
Delay caused by the physical
distance between tiers
Common p
problem in chattyy
applications
Bandwidth
Delay caused by transmission
(i.e. putting the bits onto the
wire)
Protocol
Could be non-optimal TCP
settings
Check Diagnosis tab for
bottleneckstest
Check Statistics tab for values
Congestion
Delay caused by other traffic in
the network
58
Summary of Delays: Parallel Effects

Delay incurred when multiple types of delay (such as tier processing,
bandwidth, and latency) occur simultaneously
test
59
Inc.
29
2/10/2011
AppDoctor After Network Definition (16.0 )
test
BEFORE
60
AFTER
Summary of Delays: Drill-Downs

Double-click any section of the pie chart for extra information
Drill-downs give more information that can help diagnose problems
Ex: Bandwidth drill-down shows throughput
g p graphs
g p and link bandwidth
test
61
Inc.
30
2/10/2011
AppDoctor: Executive Summary

Gives overview information for application task
Total transaction time
Total application data transferred
Single
Si l greatest
t t cause off delay
d l
Tiers in application
Bandwidth and latency at each tier
test
62
AppDoctor: Diagnosis
Displays common
application, network,
and server problems
Flags bottlenecks and

potential bottlenecks
Various characteristics
of the ACE file are
compared with
thresholds
Click on any cell

(especially on test
bottlenecks) for extra
information
63
Inc.
31
2/10/2011
AppDoctor: Statistics
Tabular view of key
application characteristics
of statistics quantify
Effect
Effect of
the components of delay
Other statistics include

Application message statistics
Network packet statistics
TCP statistics
Tip: Use it with Diagnosis
tab to
test
quantify bottlenecks
Tip: You can export to spreadsheet
64
AppDoctor: Protocol-Specific Tabs

Protocol-specific information,
statistics, and diagnoses
Appear automatically when

certain protocols are detected in
an ACE file:
HTTP
Citrix
CORBA
SOAP (Web Services)
test
65
Inc.
32
2/10/2011
Right-Click Operations in AppTransaction Xpert

Graph Statistics such as throughput or congestion
Exclude or permanently delete packets and messages
Include ppackets and messages
g
Rename a tier or lookup a tier name with DNS
Re-code traffic
Zoom in and out in the DEC
Show protocol decodes for selected traffic
Export decodes for selected frames in the decode viewer
test
66
Take-Away Points
AppTransaction Xpert provides an analysis of the main application
bottlenecks that is easy to use and to understand. The diagnosis is displayed
and broken down into components of delay. The different components are:
Processing Effects
Network Effects
Latency
Bandwidth
Protocol
C
Congestion
ti
(with recommended options)
test
Other Effects
Parallel Effects: Combination of any processing effects and network
effects that occur at the same time
67
Inc.
33
2/10/2011
Reports
test
Reporting Motivation
Quick and easy to use
Support your findings
Recording and dissemination of all
analysis results to all interested
stake holders
Documentation of application
behavior for future reference
test
69
Inc.
34
2/10/2011
Report Selection - Type and Format

Very simple and quick select report type for output (from pull down menu, or icon
button)
test
70
Report Options
Select report options and generate report
Report sections, title (statistics)
Report directory (output), Report Server (optional)
test
71
Inc.
35
2/10/2011
ACE MS Word Report
test
72
ACE Web Report

URL
Home Dir
(all reports)
test
73
Inc.
36
2/10/2011
ACE Multiple Transaction Report
Trans 1
Trans 2
test
74
ACE Difference Report
test
75
Inc.
37
2/10/2011
ACE Spreadsheet Export Reports

Comparison Report
Tree View Statistics
AppDoctor Statistics
test
76
ACE Report Template

Report template Defines the previously-defined templates used
to generate the report. You can modify and customize report templates.
The standard templates:
ApplicationDeveloperReport.ace.rt.txt
ManagementReport.ace.rt.txt
NetworkEngineerReport.ace.rt.txt
concise_report.ace.rt.txt
detailed_report_ace.rt.txt
standard_report_ace.rt.txt
Reports can be cataloged and managed by the Report Server

test
Provides security,
allows access by only registered users
Provides automatic ageing of reports
77
Inc.
38
2/10/2011
LAB 2
LAB 2
Application
Performance
Troubleshooting
test
78
Lab 2: Application Performance Troubleshooting

Overview
Response time problems for remote users of new web application
Remote site connected via a 256-Kbps Frame Relay circuit
Need to determine cause of the delay (network or web server)
Traces have been captured at both server and client
Objectives
Visualize application in ACE
Identify the cause of the application delay
Use Transaction Analysis to identify problem with application
Starts on page
9 in your lab manual
test
79
Inc.
39
2/10/2011
Lab 2: Conclusion
Conclusions
80
The web page takes approximately 24 seconds to download

Bandwidth and congestion are the biggest sources of delay
Server delay is not a problem
Majority of traffic flow is from the server to the client
Web page consumes almost all of the 256-Kbps circuit bandwidth
Largest transaction is a download of a 352 KB graphic
This single
graphic takes 18 seconds to download
test
Performance Predictive Studies

The use cases for predictive studies:
Predict application
pp
behavior under different network conditions
Problematic applications
Evaluate performance impact of application design decisions

Modified applications
Problematic application
Validate infrastructure readiness for new deployments

test
81
Inc.
40
2/10/2011
Prediction in AppTransaction Xpert

What can predictive analysis tell you?
Response time of application under different circumstances
Will addingg bandwidth help?
p
Is this application sensitive to latency?
Can we roll this out to remote users?
What sort of studies will it help with?

Troubleshooting performance problems
Testing a new application
Deploying a current application to new users
test
Server relocation studies
82
Application Deployment Workflow

Identify performance issues
Capture and import into ACE
Create report and prediction
scenario templates
Use QuickPredict to predict
performance
test
O ti
Optional:
l U
Use M
Multi-User
lti U
QuickPredict for basic capacity
planning
Optional: Use IT Guru for
network and server capacity
planning
83
Inc.
41
2/10/2011
Application Deployment Workflow

Optional: Use simulation for further studies
Discrete Event Simulation to predict response times or perform server
capacity planning
Flow Analysis for network capacity planning
Simulation covered later in course
test
84
QuickPredict
test
Inc.
42
2/10/2011
QuickPredict
Simplified simulation engine
Helps study effects of the network on the application
Q
Questions Q
QuickPredict can answer
How much bandwidth would it take to get a 4 second response time?
Would increasing the TCP Window improve response time?
What happens to response time if latency increases?
How sensitive is this application to packet loss?
What would the response time be if the link was 65% utilized by other
traffic?
What
h if I had
h d a 10% WAN
A Acceleration
A l i in
i my Networkk NEW !!
test
86
QuickPredict: Outputs
Two different output styles
QuickPredict Bar Charts shows components of delay for different scenarios
QuickPredict plots response time over range of network conditions
test
87
Inc.
43
2/10/2011
QuickPredict Bar Charts: Scenarios

Click to add multiple scenarios
Right-click to remove scenarios
test
88
QuickPredict Bar Charts: Options

Put parameters on slider
Drag slider back and forth to
dynamically update bar chart
Rename scenarios
Symmetric Bandwidth
Set asymmetric values for
bandwidth
Label delay categories
Load and save parameter settings
using templates
test
89
Inc.
44
2/10/2011
QuickPredict Bar Charts: Options NEW

Introduce WAN Acceleration
Byte Reduction
Turn Reduction
C
Compression
i
Caching
test
90
QuickPredict Bar Charts: Tabular Results

View detailed results for each scenario in the Tabular Results tab
test
91
Inc.
45
2/10/2011
QuickPredict: Response Time Graphs

Shows sensitivity of task response time to network characteristics:
Bandwidth
Latencyy
Packet Loss
Link Utilization
TCP Window Size
WAN Acceleration
test
92
QuickPredict: Advanced View

Choose network path to modify
Choose an X-axis parameter and the range of values over which to plot
response times
Select values for each of the other parameters
Click Update Graph
Overlay multiple curves on same graph
Load and save templates
test
93
Inc.
46
2/10/2011
QuickPredict: Table View

Modify parameters on all network paths in the task
Red text indicates X-axis parameter
Use Set X-axis to specify
p
y X-axis p
parameter
Load and save templates
test
94
QuickRecode
test
Inc.
47
2/10/2011
QuickRecode
Changes high level application characteristics
Shows sensitivity of task response time to application characteristics:
Application turns
Total application bytes sent
Total processing time in tiers
Use with QuickPredict to see how response times are affected
test
96
QuickRecode: Use Cases

Use Cases
Performance benefits
Changing database access
to reduce chattiness
Demonstrate effect of faster
app server
Hint: Use QuickRecode to

prove effect of optimizing
application
test
97
Inc.
48
2/10/2011
LAB 3
LAB 3
Predictive
Analysis
test
98
Lab 3: Predictive Analysis

Overview
Deploying a new application to several remote locations
Application has client querying an Oracle database
Application has been deployed to pilot site (in same city as database) and
poor performance is already apparent
Goal is to determine if the application deployment will be successful
If not, what can be done to improve performance?
Objectives
Perform a deployment study using QuickPredict Bar Charts
Use AppTransaction Xpert to drill down into the app and see whats going
on
test
Use QuickRecode
to test a possible fix
99
Inc.
49
2/10/2011
Lab 3: Conclusion
Always try to determine goal or objective of any problematic application
Is there an SLA expectation
What is the user expectations
What are other users, departments, experiencing
Additional useful troubleshooting information to determine

Is the problem reproducible or is it transient
Are all transactions of the application behaving the same
test
100
Agenda
Capture Methodologies
Capture Data Sources
- New Capture Manager Options
Usingg AppTransaction
pp
Xpert
p
to p
perform p
packet captures
p
- Capture Types
- Lab 4: Capture Process
Using AppTransaction Xpert to filter packet captures

- Capture Filters
Using AppTransaction Xpert to import packet captures

- Import Process
- Lab
L b 55: Identifying
Id tif i P
Performance
f
P
Problems
bl
in
i Unfiltered
U filt d Traces
T
test
101
Inc.
50
2/10/2011
Capture Methodologies
test
Basic Capture Methodology

Define and understand the problem at hand
What is slow?
Is it the login or another specific task?
How often does it occur?
What is a task?
Isolate an atomic Transaction
Loading single web page
From when user clicks Submit to screen refresh
Plan your capture

Simultaneous or single-sided
Which tiers are involved in the application?
Aim for as clean

test a packet trace as possible
103
Keep measurement as simple as possible

Filter out unrelated traffic
Capture one occurrence of one task at a time
Perform capture numerous times to help eliminate anomalies
Inc.
51
2/10/2011
Atomic Transaction Example (Task)

Web Application of a Bank
Load Login Page
Open a Web Browser
Type
yp URL of the Bank
Enter
Login Page Loads
Login
Type User credentials
Login
Home Page shows up
Check Savings Account
Mouse/cursor around
Click on Savings
Savings Account
Account
Savings Accounts
Page Loads
test
Logout
Call up Logout screen
Submit
Logged out page
104
Red arrow = Start Capture

Blue arrow = Stop Capture
test
Inc.
52
2/10/2011

Packet traces are the primary data inputs
AppTransaction
pp
Xpert
p
supports
pp
a wide variety
y of packet
p
sources
ACE Capture Agents

ACE Live
ACE Enterprise Management Server
Agentless Capture Options NEW!
F5 Load Balancers
UNIX, Linux platforms
WAN Accelerators
test
Cisco NAM
Wireshark and other Winpcap and Libpcap sources
TCPDump, Snoop
106
Agentless Capture Options NEW

No software installation required
Use the AppTransaction Xpert Capture Manger to control captures remotely
via SSH
Supported on platforms that support TCPDump
TCPDump file remains on machine until capture is completed, then deleted after
retrieval
Included in the AppTransaction Xpert Plus license
test
107
Inc.
53
2/10/2011
Agentless Capture Options NEW (cont.)

F5 Load Balancers
Excellent way to follow a transaction to the proper server
Perform successful trace file merging across firewalls
Rule out the device as a source of delay
UNIX-Based Operating Systems

Supported Platforms
Linux
Solaris SPARC
Solaris Intel/AMD
AIX
FreeBSD
test
HP-UX Itanium
Macintosh PowerPC
Macintosh Intel
True64
108
Agentless Capture Options NEW (cont.)

WAN Acceleration Devices
Support for Riverbed and Cisco
Restore visibility of the WAN
More information can be found in session 1471 - Monitoring and
Troubleshooting Applications in a WAN Accelerated Environment with
ACE Live and AppTransaction Xpert
test
109
Inc.
54
2/10/2011
Capture Data Sources AppResponse Xpert

(ACE Live)
New seamless 1-click integration
Let ACE Live do the filtering for you
Easily pull traces from alert snapshots

off actual
t l application
li ti issues
i
test
110
Capture Data Sources AppForensics Xpert

(ACE EMS)
ACE Enterprise Management Server
Provides web-based extension to Continuous Capture functionality

Supports both End-User Driven Capture workflows, as well as Server-Driven
Capture Workflows
test
End Users report application slowness

Trouble ticket is created with trace files of the incident attached
Troubleshooters claim problem reports and troubleshoot on their schedule
111
Inc.
55
2/10/2011
AppForensics Xpert Web Capture Retrieval
test
112
Capture Data Sources AppForensics Xpert

Server-Driven Capture Workflow
Panorama provides forensic actions based on threshold violations within the
application servers
The trace file forensic action automatically generates a trace that is accessed
through AppForensics Xpert
For more detail, attend session 1482 Deploying an Enterprise-Wide Packet
Capture Architecture with ACE EMS and ACE Live
test
113
Inc.
56
2/10/2011
Capture Data Sources ACE Agents

Benefits of using ACE Agents
Unlicensed
Support for major platforms (Windows, HP-UX, AIX, Solaris, Linux, Mac OS X,
FreeBSD)
Lightweight, Secure, and Stable
Silent Install capability
Server Statistics collection (Windows)
Enhanced Filtering (NEW)
On-Demand and Continuous Capture modes
test
114
Additional Data Sources

AppTransaction Xpert can import a variety of additional data
Identify these sources during capture to ensure collection
From OPNET products

Windows ACE Capture Agents can collect Lightweight Server Statistics
OPNET Panorama
Server statistics and performance metrics
Java transaction tracing
Third-party sources
HP GlancePlus
Perfmon
Custom datatest
in comma
saved (.csv) format
115
Inc.
57
2/10/2011
Capture Lightweight Server Statistics

Capture Windows performance data
ACE Capture Agent generates two files
Application traffic (*.appcapture) file
Windows performance data (*.appcapture.da) file
Display internal server metrics

Percent Processor Time
File Read Bytes Per Second
System Calls Per Second
Available MB
File Write BytestestPer Second
116
AppInternals Xpert (Panorama) Server Statistics

Capture Server Statistics from AppInternals Xpert
Use with on-demand captures
Thousands of available metrics
For more information:

Session 1461- OPNET Panorama Essentials: Multi-Tier Application Monitoring,
Correlation, and Troubleshooting
test
117
Inc.
58
2/10/2011
AppInternals Xpert Transaction Trace

AppInternals Xpert can create a trace of all activity through the Java
Virtual Machine (JVM) or .NET CLR
Provides a call-by-call sequencing (or call stack) of all activity within your
application
li i
This transaction trace can be imported into AppTransaction Xpert to explain

what is occurring on a given system during long processing delays
test
118
Capture Manager
Capture Manager control parameters
test
119
Inc.
59
2/10/2011
Capture Agent Settings
test
120
Capture Types
test
Inc.
60
2/10/2011
On-Demand Capture
Methodology
User knows exactly when a problem is going to occur and
what it takes to reproduce it
Deploy and configure ACE Capture Agents
Start the capture
User re-creates the problem
Stop the capture
Import capture for analysis
Import
test
122
On-Demand Capture (cont.)
test
123
Inc.
61
2/10/2011
On Demand Capture Process: Adding an Agent
test
124
On Demand Capture Process: Specify Agent

Settings
test
125
Inc.
62
2/10/2011
Capture Process: Start Capture
test
126
Capture Process: Finish Capture
test
127
Inc.
63
2/10/2011
Capture Process: Open in Transaction Analyzer
test
128
Scheduled Capture
Useful for reproducible problems during off hours
Methodology
User knows when a problem will occur
For example, after-hours batch process or database backup
Schedule a capture
Choose one-time or repeating capture
Set the schedule for a capture
Specify schedule for download
Import
I
t trace
t
for
f analysis
l i
test
129
Inc.
64
2/10/2011
Scheduled Capture (cont.)
test
130
Scheduled Capture (cont.)

Scheduled capture can create one-time,
daily, weekly, or monthly capture
schedules
test
Scheduled capture utility can optionally

create a one-time, daily, weekly, or
monthly schedule to download trace files
131
Inc.
65
2/10/2011
Continuous Capture
Useful for troubleshooting intermittent problems
Methodology
User experiences intermittent problem
Start continuous capture
User calls help desk
Problem occurred again around 9:30
Preview capture
Download
Download trace of problematic time span
test
Import trace
for analysis
Agent details
Same capture agents as on-demand capture
132
Continuous Capture (cont.)
test
133
Inc.
66
2/10/2011
Continuous Capture: Preview/Download
test
134
LAB 4
LAB 4
Capture
Process
test
135
Inc.
67
2/10/2011
Lab 4: Capture Process

Overview
Configure a capture agent
Perform an on-demand capture
p
Open in Transaction Analyzer
Skills covered
Learn how to configure a capture agent
Understand how to use agent filters
Capturing from the network
test
136
Lab 4: Conclusion
Configured capture agent for capture
Located and specified host IP Address
Specified capture mode
Set up a filter
Performed a capture
Started the agent
Performed a transaction
Stopped the agent
Brought
g the capture
p
file into Transaction Analyzer
y
test
137
Inc.
68
2/10/2011
Capture Filters
test
Filtering Unrelated Traffic

You want to capture the transaction of interest
AppTransaction Xpert lets you filter out irrelevant traffic
Options to filter during capture, during the import or after the import
The earlier the better
Why filter during capture?

Only capture the specific transaction
Smaller trace file to process => easier/faster import
Why not filter during capture?

Sometimes you want to see the whole picture
If youre not 100% sure about application path
Recommended
test methodology
Capture without filters for short duration

Examine trace
Determine best filter to use
Capture subsequent tasks with filter
139
Inc.
69
2/10/2011
Filter: During Capture

Users can filter out or only allow traffic based on a specific protocol
Users can filter specific hosts

and/or ports for incoming or
outgoing traffic
New Filtering Options in

Capture Agent test
3.9
140
Use wildcards and slash notation

Use DNS names
Port ranges
Broadcast and multicast can be
excluded independently
Users can elect to exclude or

allow broadcast and multicast
traffic
Filter: During Capture (cont.)

Preview in Continuous Capture
Select the IP addresses, conversations and time ranges of interest
Great for troubleshooting intermittent issues
test
141
Inc.
70
2/10/2011
Filter: During Import

What is Trace Explorer?
Graphical protocol analyzer
Used to helpp filter unrelated traffic
during import process
Why would you use it?

Graphically filter a capture file
Allows for fast read of large trace
files
You want to know what youre
looking at
test
142
Filter: After Import

Once capture file has been imported, users
can still filter irrelevant traffic
Right-click on any tier pair

conversation, application message,
application
li i transaction
i or networkk
packet
Option to Exclude or Permanently
Delete
Filter Wizard
Remove unidirectional, non-IP, UDP,
broadcast traffic
Capture File Filter

Use to remove traffic by host, port,
protocol
143
test
Inc.
71
2/10/2011
Import Process
test
Import Overview
Now that the data has been captured, it must be imported into
Review what has been captured

Do you have a single capture file?
Do you have more than one capture file?
Are you capturing WAN accelerated traffic?
Is the capture file filtered?
If not, use Trace Explorer
Greatly simplified
for 16.0 !!
test
145
Inc.
72
2/10/2011
Import Changes in 16.0

Import process has been greatly simplified
15.0 required answering a series of questions about tier locations, bandwidths, and
latencies
For 16.0, we focused on asking as few questions as possible
Filter, view statistics or diagnoses, or explore your transaction immediately
Locations, bandwidths, and latencies can still be specified to break down Network
Transfer effects into Bandwidth, Protocol, and Congestion
Most importantly, you can change these values whenever you need to
Import is now broken down into use cases

Single packet trace
Merge several traces
Merge traces
from a WAN-accelerated environment
test
Batch import
Filter into Trace Explorer
146
Import a Single Capture File
test
147
Inc.
73
2/10/2011
Merging Capture Files

What is a merge?
Two or more traces of an application transaction can be combined
Creates unified view of application
Accurately determines the send and receive times of each packet
Benefits and use cases

Allows analysis of multi-tier transactions where all traffic does not
go through one location
Improves analysis in a production environment with congestion issues such as
packet loss and retransmissions
test
148
Merging Capture Files (cont.)

All traces must be of the same transaction
All must have been captured at the same time
Easier to import
p if there is some common traffic between traces
test
149
Inc.
74
2/10/2011
File name
test
Load previously saved import

details (tier names, tier locations,
bandwidth, latency,)
150

Replace IP addresses with friendly names (can use DNS)
test
151
Inc.
75
2/10/2011
After Opening/Merging
Now what?
Filter down to a single transaction or remove user think time
Open AppDoctor
Note that there are only Processing and Network effects identified
Specify bandwidth and latency to break out individual network effects
test
152
AppDoctor
Bandwidth, protocol and

congestion are all
included in one category
named Network
Transfer
test
153
To break down the various

components
t off Network
N t
k
Transfer, click the
Refine Network
Effects button
Inc.
76
2/10/2011
Refine Network Effects
Specify the local and

remote bandwidths
here
test
154
Tier locations and

latencies are
automatically
detected, but can be
changed if needed
Specify Bandwidth and Latency

Minimum bandwidth
Capacity of the slowest link
Should be a known factor about your network
Minimum one-way latency
PathProbe provides an estimate

Half the value returned by ping
Auto-calculated in a trace merge
In manyy cases,, AppTransaction
pp
Xpert
p
can estimate latencyy of a single-sided
g
trace
test
155
Inc.
77
2/10/2011
Specify Bandwidth and Latency (cont.)

Quiz
Bandwidth of each segment is known
Ping reports 46ms RTT
test
What would you specify for bandwidth and latency between the Client and
the Server during import into AppTransaction Xpert ?
Bandwidth = 1544 Kbps

156
Latency = 23ms
AppDoctor After Network Definition
test
BEFORE
157
AFTER
Inc.
78
2/10/2011
LAB 5
LAB 5
Identifying
Performance
Problems in
Unfiltered Traces
test
158
Lab 5: Identifying Performance Problems in

Unfiltered Traces
Overview
Open a single side capture
Filter traffic
Refine network effects
Analyze transaction
Skills covered
Step through the import process in AppTransaction Xpert
Understanding the importance of bandwidth settings
test
159
Inc.
79
2/10/2011
Lab 5: Conclusion
Imported a single-sided capture file
Imported capture file
Filtered irrelevant traffic
Estimated locations
Entered local and remote bandwidth
Estimated latency
test
160
Introduction to AppTransaction
Xpert
test
Day 2
Inc.
80
2/10/2011
What to Expect in Advanced AppTransaction

Xpert Topics
Mix of lecture and lab
Advanced class
Address network concepts important to using AppTransaction Xpert
Dive into the science behind ACE
Highlight new ACE features
Discuss TCP
Please be interactive
Participate test
Ask questions
162
Prerequisites
Prerequisites
Reviewed yesterdays material
Reviewed session 1441
Equivalent experience
You should already understand the mechanics of ACE

Capturing a trace
Importing traces
Generating reports
You should already

recognize vanilla problems
test
Chatty applications
Insufficient bandwidth
Slow servers
163
Inc.
81
2/10/2011
Agenda
How AppTransaction Xpert Calculates and Predicts Components of Delay
- Lab 1: Calculating Network Delays
Understanding the different Components of Network Delay

- Congestion Delay
- Protocol Delay
- Lab 2: Understanding Components of Delay
Understanding Parallel Effects

- Parallel Effects
- User Think Time
test
- Lab 3: Understanding Parallel Effects
164
How AppTransaction Xpert

calculates and predicts
p
of Delayy
components
test
Inc.
82
2/10/2011
Components of Application Delay

The following are key components of delay in a transaction:
Processing
Client
Server(s)
Network
Protocol (TCP WIN, Nagles, etc.)
Bandwidth
Latency (baseline, dominated by network hardware and ultimately ruled by C)
Congestion (including queuing)
Packet loss
Application itself
test
Think of the top two topics as resources. How does the app use them?
Efficient/inefficient code
App turns/Chattiness
166
Review of Network Delays

Packets experience a complex series of
Network hops
Technologies
Queuing delays
T1
OC3
test
167
Inc.
83
2/10/2011
Apps See the Network as a Black Box, So Does

For most troubleshooting, complexity can be simplified
Think of entire network as a single equivalent circuit
Once packet leaves source, the delay can be separated into 3 categories
Bandwidth delay
Latency delay
Congestion (queuing) delay
What about Protocol?
Protocol delay is a function of the sender/receiver
T1
OC3
Packet Loss is handled by the protocol
test
Bandwidth, Latency,
Congestion
168
Bandwidth Delay
Time it takes to stream all bits onto circuit
This number varies with packet size
Can be calculated from ppacket size and circuit bandwidth
2000 bits
Headers
Data
2000 bits/second
01110010010
01110010010
test
What is the bandwidth delay for this packet?
169
Inc.
84
2/10/2011
Calculate Serialization Delay

How long does it take
to serialize this packet
on to the wire?
Serialization and BW
are the same thing
If you know the bottleneck bandwidth across a link and the size of each
test
packet, you know how long it takes to get across the link due to serialization
delay (ie: Bandwidth delay)
Note: we are ignoring latency at this point
170
Latency Delay
Time it takes for signal (1st bit) to travel across network
Distance / speed of light
Normal delays caused by device forwarding
This is a constant number for all packets

7 ms
1st bit
10 ms
1 ms
10 ms
1 ms
10 ms
1 ms
What is the latency

for this circuit?
test
What value would ping produce?
171
Inc.
85
2/10/2011
Application Turns
A turn is a change in the flow direction of application data
Turns magnify circuit latency
An application
pp
with no turns would onlyy experience
p
latency
y once
An application with 99 turns would experience latency 100 times
Simple equation for total latency delay
Latency Delay = circuit latency * (turns +1)
This equation only works on sequential applications
No Application
pp
Turns
Manyy Application
pp
Turns
test
172
Congestion Delay
Queuing on devices
Usually caused by other traffic
Can varyy from packet
p
to packet
p
Forwarding rate = 1 packet / ms
packet
test
5 packets in forwarding queue
What is the congestion delay for this device?

What is the total delay across this device?
173
Inc.
86
2/10/2011
Determine Queuing/Congestion
Packet should have taken

thi time
this
ti . . .
. . . but ended up
taking this time
test
If a packet should take x time but takes y time, y x yields

queuing/congestion delay
174
Example Network Delay

Bandwidth = 1,000 bps
Latency = 5 seconds
T=0
10
20
30
40
50
Client
Bandwidth delay
30 seconds
30,000 bits
Latency delay
5 seconds
Congestion delay
test
Server
15 seconds
50
175
Inc.
87
2/10/2011
Packet Trains
Stream of packets that are sent together
Arrive at destination spread out
Manyy TCP segments
g
of a larger
g application
pp
message
g
Spreading caused by limited bandwidth of bottleneck link
T=0
10
20
30
40
50
60
10
20
30
40
50
60
test
0
176
Calculating Delay for Packet Trains

Treat packet trains like 1 big packet
Same rules apply
Firstestablish delay caused by bandwidth (stream bits onto wire)
Secondadd latency (last bit of the packet train getting across)
Thirdanything left over is congestion
Application Layer
Transport Layer
test
Network Layer
177
Inc.
88
2/10/2011
Calculating DelaysAdvanced
Bandwidth = 1,000 bps
Latency = 5 seconds
T=0
10
20
30
40
50
60
70
80
90
100
Client
Bandwidth Delay
5 + 10 + 30
Latency Delay
5 +5 +5
Congestion Delay
5 + 15
Tier Processing (Client)
10
test
10
5,000 bits
20
Server
65
80
Tier Processing (Server)

10
100
10,000 bits
Total Delay = 45+15+20+10+10 = 100

178
Pie Chart
AppTransaction Xpert creates a pie chart that shows all of the components of delay in one
screen
Shows the total impact on response time due to each individual effect
Each wedge is the maximum benefit that a single upgrade will provide
Bandwidth
45
Bandwidth
Latency
15
Latency
Congestion
20
Client
10
Server
10
Congestion
Client
Server
test
179
Inc.
89
2/10/2011
Network Effects in App Doctor

AppTransaction Xpert 16 does not require fully specifying network data for
import
Creates a Network Effects pie slice
test
180
Network Effects
Specifying Bandwidth will cause AppTransaction Xpert to recalculate the
values for the components of delay
test
For more information, see

Session 1442: Application Capture
and Import Strategies with AppTransaction Xpert
181
Inc.
90
2/10/2011
Calculating Delays
Two ways to calculate the Pie Chart
Adding up (add the components of delay)
Squeezing down (reduce effects one at a time)
T=0
10
20
30
40
50
60
70
80
90
100
Client
test
Server
10
182
20
65
80
100
Review of Network Delays

Different factors can affect network delay
Bandwidth
Latency
Congestion
Some factors are constant and some are variable

Network delay is not always due to lack of bandwidth
Important that you understand fundamentals to interpret ACE results
test
183
Inc.
91
2/10/2011
LAB 1
LAB 1
Calculating
Network
Delays
test
184
Lab 1: Calculating Network Delays

Overview
Apply knowledge of sources of delay to a real application
Objectives
From the raw statistics calculate total bandwidth delay
From the raw statistics calculate total latency delay
Compare results with ACE analysis
test
185
Inc.
92
2/10/2011
Lab 1: Conclusion
Response Time = 26.03
Network Data = 3,199,760 Bytes
Size of link = 1,544,000 Kbps
p
Bandwidth Delay = (3,199,760*8)/1,544,000 = 16.58
Percentage of Bandwidth = 16.58/26.03 = 63.7%
Turns = 337
Latency = 24.23ms
Latency Delay = 24.23*(337+1) = 8.18 seconds
Percentage
P
t
off L
Latency
t
= 8.18/26.03=
8 18/26 03 31.4%
31 4%
test
186
Lab 1: Conclusion
Calculated that Bandwidth caused 63.7% of response time
Calculated that Latency caused 29.8% of response time
test
187
Inc.
93
2/10/2011
Lab 1: Conclusion
Why didnt the math work out perfectly?
Some amount of congestion in trace
Some amount of protocol delay in trace
Equation
E ti didnt
did t accountt for
f bi-directional
bi di ti l traffic
t ffi
Our simple math was very close

Important to perform this type of sanity check
test
188
Congestion Delay
test
Inc.
94
2/10/2011
Congestion Delay
It is somewhat amazing that congestion can be calculated
Very difficult to measure and calculate manually
ACE can measure the effect even without seeing the cause
ACE provides 2 important pieces of information for
congestion
Actual congestion delay for each packet
The cumulative effect of congestion on the response time
test
190
Key Concept
Trace file includes 1 piece of time information
Time the packet arrived at the capture probe
1 piece of time information per packet
ACE includes 2 pieces of time information for each packet

Time the packet left the sending side
Time the packet arrived at the receiving side
test
2 pieces of time information per packet

191
Inc.
95
2/10/2011
Key Concept
ACE knows both the send and receive time
Delay analysis requires that both values are known
How does ACE derive the second ppiece of time information?
Trace Merge
Single Trace Adjust
test
192
Trace Merge
Each trace shows what happened locally
Once clocks are aligned, draw the packets from send to receive
Trace captured at client
side
T = 0 1 2 3 4 5 6 7 8 9 10
Client
T = 0 1 2 3 4 5 6 7 8 9 10
Client
BC
C*
B C
C*
Trace captured at server

test
side
AB
C* D
A B
Server
C* D
Server
0 1 2 3 4 5 6 7 8 9 10
0 1 2 3 4 5 6 7 8 9 10
193
Inc.
96
2/10/2011
Single Side Adjust

With a single trace, ACE only knows what happened locally
Must estimate what happened on network and at other side
Based on bandwidth and latency values set during import
ACE estimates
ti t when
h th
the packet
k t was sentt
ACE estimates when the packet will arrive
T = 0 1 2 3 4 5 6 7 8 9 10
Client
A B
C*
Trace captured at client

side
T = 0 1 2 3 4 5 6 7 8 9 10
Client
BC
test
C*
Server
0 1 2 3 4 5 6 7 8 9 10
194
Trace Merge
What happens if the import parameters are incorrect
Bandwidth is set too low
g
Latencyy is set too high
Might create false packet crossings

This will confuse analysis
T = 0 1 2 3 4 5 6 7 8 9 10
Client
A B
Trace captured at client side

T = 0 1 2 3 4 5 6 7 8 9 10
Client
BC
test
Server
0 1 2 3 4 5 6 7 8 9 10
195
Inc.
97
2/10/2011
Key Concept
With BOTH send and receive time ACE knows
Actual packet delay
packets were sent and received
Order of how p
What would happen if we graphed the delays?
testTotal
196
Packet Delay
Packet Size vs. Delay Graph
Delay (sec)
Graph the packet size vs. total packet delay
Congestion
Slope is bandwidth*
test
Pure latency
Packet Size (bits)

* Inverse of bandwidth (1/BW)
197
Inc.
98
2/10/2011
Packet Size vs. Delay Graph (cont.)
Delay (sec)
Graph Packet Size vs. Total Packet Delay
test
Packet Size (bits)
198
Viewing Congestion in ACE

Several good ways to view congestion in ACE
AppDoctor diagnosis
Summary of Delays
Congestion drill-down
Data Exchange Chart
test
199
Inc.
99
2/10/2011
Congestion in the AppDoctor Diagnosis

Specifically marks category as a Bottleneck
test
200
Congestion in the Summary of Delays

Shows the cumulative effect of the congestion on response time
How much slower was the application because of congestion?
test
201
Inc.
100
2/10/2011
Drilling into Congestion
Drilling into congestion can show if the packets are getting queued or
test
otherwise slowed down
202
Drilling into Congestion (cont.)
Look for plateaus

test
Always keep in mind the scale of the chart!
Is delay uniform or random?
Is there any encryption on the link?
203
Inc.
101
2/10/2011
Congestion in the Drill-Down

Packet Congestion graph
Retransmission graph
test
204
Congestion in the DEC

Network chart layered with Packet Congestion graph
test
205
Inc.
102
2/10/2011
Congestion Review
Congestion Delay is cause by other traffic on the network
Always better to merge traces from congested circuits
Congestion values for a trace merge are precise
Congestion values for a single-side adjust are approximations
Several ways to see congestion and its effect on the application

Imagine squeezing the congestion out of the trace to calculate its
contribution to overall delay
test
206
Protocol Delay
test
Inc.
103
2/10/2011
Protocol Delay
Caused by the artificial throttling of traffic
Packets are prevented from even getting onto the network
Caused mainlyy byy TCP
test
208
Why Does TCP Exist?

TCP offers reliable data transport to applications
Application data is guaranteed to be delivered
Packets that are lost by the network will be retransmitted
TCP makes applications behave in a network-friendly manner

Throttles the rate at which application sends data
Ensures that no single user consumes all of the bandwidth
Both can impact application performance
test
209
Inc.
104
2/10/2011
What Does Protocol Delay Look Like?

Protocol delay prevents packets from even getting on the network
Application Layer
Transport Layer
Network Layer
No Protocol Delay
Protocol Delay
test
Protocol
Delay
0 1 2 3 4 5 6 7
210
0 1 2 3 4 5 6 7 8 9 10 11 12 13
Observe Protocol Behavior
Protocol delayy is when the

sending side delays or
throttles sending of data (this
is different than processing
delay)
Observe how data leaves in stages indicating Protocol Delay

test
Observe other attributes like TCP window size
Our example here is ridiculously simplified, but real protocol delay is
observable in a real trace
211
Inc.
105
2/10/2011
Forms of Protocol Delay

Critical that you understand mechanisms of TCP
Several different TCP mechanisms cause can protocol delay
TCP windowing
Slow Start
Frozen window
Nagles algorithm
Retransmissions
Out of sequence packets
test
212
Viewing Protocol Delay in ACE

Several ways to view Protocol Effects
AppDoctor diagnosis
Summary
S
off D
Delays
l
Protocol drill-down
TCP Visualization in DEC
test
213
Inc.
106
2/10/2011
Protocol Delay in the AppDoctor Diagnosis

Specifically marks category as a Bottleneck
test
214
Protocol Delay in the Summary of Delays

Double-click on slice to get more detail
test
215
Inc.
107
2/10/2011
Protocol Delay Drill-Down

Graphs dynamically appear based on AppDoctor Diagnosis
Users can manually add or remove graphs
Protocol Delayy
TCP In-Flight Data
Retransmissions
Out of Sequence Packets
TCP Nagles Delay
TCP Advertised Receive Window
test
216
Protocol Delay Review

Protocol delay results from the artificial throttling of traffic
Several different TCP mechanisms can cause it
Prevents the ppacket from even getting
g
g onto the network
Can often be calculated without a trace merge
Imagine squeezing the protocol effects out of the trace to calculate its
contribution to overall delay
test
217
Inc.
108
2/10/2011
LAB 2
LAB 2
Understanding
Components of
Delay
test
218
Lab 2: Understanding Components of Delay

Overview
User is transferring files with Microsoft file browser
Transfer time is very slow
Want to understand cause of delay
Objectives
Identify and understand the sources of delay
Compare slow transfer to fast transfer

Homework
test
Continue with homework sections of lab manual if you have time
219
Inc.
109
2/10/2011
Lab 2: Conclusion
File transfer is slow
Primary source of delay is protocol/congestion
Some potential problems with TCP but no smoking gun
Look at packet delay graph to evaluate congestion
Significant congestion on WAN circuit
Base latency is 25ms
Some packets take 125 ms to cross the network
test
220
Lab 2: Conclusion
Congestion is primary cause of delay
Removing other traffic on network fixes problem
Reduces congestion
Response time decreases
Might be a case for QoS
Significant Congestion
No Congestion
test
221
Inc.
110
2/10/2011
Parallel Effects
test
The Mysterious Gray Wedge
AppTransaction Xpert shows parallel effects as a gray pie wedge

Transaction has more than one effect taking place at one time
test
HTTP is an example of a transaction that exhibits parallelism
223
Inc.
111
2/10/2011
Parallel Effects Drill-Down

Can show the individual effects with their combined parallel effects together
Solid bars are the individual effects
Striped
p bars are the parallel
p
effects
Optimizing both components simultaneously will give the benefits of the
individual effects AND the parallel effects
T=0
10
20
30
40
50
60
70
80
Bandwidth
60%
Latency
10%
Server
20%
Parallel
10%
Response Time
90
100
Parallel Effects of Bandwidth and Latency
test
Parallel Effects of Server and Latency
224

Parallel delays may seem counterintuitive at first
It is a real effect and does happen in the real world
Real Life Parallel Effects example
p that yyou will remember!
Reading a paper while eating breakfast
Cooking a meal (dessert in the oven while steak is on the grill)
Would be difficult to split these into simple 2 category pie charts

Incorrect to say grilling steak was 50% and dessert was 50% ?
Suggests that eliminating dessert would cut dinner preparation in half
Probably more like 25% steak, 25% dessert, 50% both
Life is parallel
test
225
Inc.
112
2/10/2011
Types of Applications
Applications fall into two main categories
Transactional
Strong
St
cause and
d effect
ff t relationship
l ti hi bbetween
t
packets
k t
Each application message is dependent on the one before it
Examples
Database queries
Web transactions
File transfers
Asynchronous
No strong
cause and effect relation between packets
test
Examples
Voice
Citrix
226
Transactional application can be further divided
Sequential
Single
Si l execution
ti path
th
Application is only doing one thing at a time
Parallel
Two or more simultaneous execution paths
Application is doing multiple things at a time
test
227
Inc.
113
2/10/2011
Sequential (Adding up method)
Each slice of time can be placed into a single delay category
The components of delay add up to the total response time
T=0
10
20
30
40
50
60
70
80
90
100
Client
test
Server
228
Parallel (Squeezing down method)
Some slices of time could be placed in multiple delay categories
The components of delay would up to more than the total response time
T=0
10
20
30
40
50
60
70
80
90
100
Client
test
Server
Connection 1
Connection 2
Connection 3
229
Inc.
114
2/10/2011
AppDoctor Individual Effects

Summary of Delays for a sequential transaction
Response Time
6 sec
Server
2 sec (33%)
Client
2 sec ((33%))
Bandwidth
2 sec (33%)
test
230
AppDoctor Parallel Effects

Summary of Delays for a parallel transaction
Server is now processing while simultaneously sending the large message
test
231
Inc.
115
2/10/2011
AppDoctor Parallel Effects (cont.)

How does AppDoctor handle multiple effects occurring at the same time?
test
The "Parallel Effects" category
Response
Time
6 sec
Server
2 sec
(33%)
Client
2 sec
(33%)
Parallel
Effects
2 sec
(33%)
Mixed wedge shows total impact due to server processing and bandwidth occurring in parallel
Because there may be many mixed wedges, ACE shows these as gray in the Pie Chart
232
Understanding Parallel Applications

Calculating delays for sequential transactions is fairly easy
Each slice of time can be attributed to a single cause
All the slices add up to the total
Calculating delays for parallel transactions can be complex

A slice of time may be attributed to multiple sources
Only the squeezing method works here
Parallel Effects are subtle and not always easy to visualize

Parallel Effects do not always occur just because delays overlap
Arent you glad you have AppTransaction Xpert to do this for you
test
233
Inc.
116
2/10/2011
Parallel Effects Example

Parallel
Two independent connections occur in parallel
Both must complete for the application to complete
Client
Conn. B
B
A
Conn. A
test
Server
234

Slide Conventions
Show independent connections as parallel DECs
Keep in mind that these are happening on top of each other
Obvious now why you cant just add delays up
T=0
10
20
30
40
50
60
70
80
90
100
Client
Connection A
Server
T=0
10
20
test
30
40
50
60
70
80
90
100
Client
Connection B
Server
235
Inc.
117
2/10/2011

Parallel applications with squeezing method
Perform experiments that squeeze out each category of delay
Color associated with that delay would disappear
Response time of transaction may decrease
Record the difference from each experiment
T=0
10
20
30
40
50
60
70
80
90
100
Client
Server
T=0
10
20
test
30
40
50
60
70
80
90
100
Client
Server
236

What would happen if bandwidth became infinite?
Green would be squeezed out
Total response time of the transaction would change to 40 seconds
Bandwidth is responsible for 60 seconds of response time (60%)
T=0
10
20
30
40
50
60
70
80
90
100
Client
Server
T=0
10
20
test
30
40
50
60
70
80
90
100
Client
Server
237
Inc.
118
2/10/2011

What would happen if latency became zero?
Red would be squeezed out
Latency is responsible for 10 seconds of response time (10%)
T=0
10
20
30
40
50
60
70
80
90
100
Client
Server
T=0
10
20
test
30
40
50
60
70
80
90
100
Client
Server
238

What would happen if the server became infinitely fast?
Yellow would be squeezed out
The server is responsible for 20 seconds of response time (20%)
T=0
10
20
30
40
50
60
70
80
90
100
Client
Server
T=0
10
20
test
30
40
50
60
70
80
90
100
Client
Server
239
Inc.
119
2/10/2011

Something interesting just happened
Bandwidth caused 60%, Latency caused 10%, Server caused 20%
The individual sources of delay are only responsible for 90%
What caused the remaining 10%?
Delays that are not removed by a single source are Parallel Effects
Bandwidth
60%
Bandw idth
Latency
test
240
Server
Parallel
Latency
10%
Server
20%
Parallel
10%

Squeeze out 2 effects at the same time
Bandwidth = 60%
Latency = 10%
Server = 20%
Parallel Effects = 10%
T=0
10
20
30
40
50
60
70
80
90
100
Client
Thread A
Server
T=0
10
20
test
30
40
50
60
70
80
90
100
Client
Thread B
Server
241
Inc.
120
2/10/2011

Remove the Server and Latency together
Server by itself is 20%, Latency by itself is 10%
The pair is responsible for 35 seconds of response time (35%)
Parallel interaction of the 2 provides a bonus benefit if changed simultaneously
T=0
10
20
30
40
50
60
70
80
90
100
Client
Thread A
Server
T=0
10
20
test
30
40
50
60
70
80
90
100
Client
Thread B
Server
242

Remove Bandwidth and Latency together
Bandwidth by itself is 60%, Latency by itself is 10%
The pair is responsible for 75 seconds of response time (75%)
Parallel interaction of the 2 provides a bonus benefit if changed simultaneously
T=0
10
20
30
40
50
60
70
80
90
100
Client
Thread A
Server
T=0
10
20
test
30
40
50
60
70
80
90
100
Client
Thread B
Server
243
Inc.
121
2/10/2011
Parallel Effects Drill-Down
Can show the individual effects with their combined parallel effects together
Solid bars are the individual effects
Striped bars are the parallel effects
Optimizing both components simultaneously will give the benefits of the individual effects
AND the parallel effects
T=0
10
20
30
40
50
60
70
Parallel Effects of Bandwidth and Latency
80
90
60%
Latency
10%
Server
20%
Parallel
10%
100
Response
Time
Parallel Effects of Server

test and Latency
244
Bandwidth
Parallel Effects Summary

Delays that can not be removed by a simple change of one parameter
To reduce, you must change at least two parameters simultaneously
These delays
y are not bad
Application is efficient
Application is doing multiple things at the same time
test
Bandw idth
Latency
Server
Parallel
245
Inc.
122
2/10/2011
User Think Time
test
Capturing a Good Transaction (Task)

Important to get an atomic transaction
Atomic transaction is the smallest transaction that you can capture effectively
Capture should be from edge of user input to screen refresh
Eliminate PEBCAK
test
247
Inc.
123
2/10/2011
Atomic Transaction Example

Modify Customer Record
Login
Call up the Login screen
Type information
Submit
Look up record
Call up correct screen/menu
Type information
Submit
Modify data
Mouse/cursor around
Type information
test
Submit
Logout
Call up Logout screen
Submit
248
End user complains

modifying a customer
record is slow
Need to find out what is
involved in transaction
Red arrows indicate the key part

of the transaction to capture
Identifying and Removing

User Think Time
Always get as close as you can to a perfect Atomic transaction as described
in the Capture
p
Section
However, sometimes you just cant get the perfect capture
User think time enters the trace
AppTransaction Xpert deals with think time
Auto detects when calling up AppDoctor
Allows user to explicitly specify think time
test
249
Inc.
124
2/10/2011
Removing User Think Time

AppTransaction Xpert automatically identifies that the trace contains user think
time. This dialog comes up when AppDoctor is selected
also allows the user to
test
pick any dependency on
the client (first talker)
and categorize it as user
think time
250
Removing User Think Time

Original Analysis
test
User Think Time

Removed
251
Inc.
125
2/10/2011
LAB 3
LAB 3
Understanding
Parallel Effects
test
252
Lab 3: Understanding Parallel Effects

Objectives
Explore an application transaction that has significant parallel effects
Trace is fairly simple
Trace is somewhat artificial
Practice visualizing the squeezing method to understand the behavior
Use Drill-down and QuickPredict to understand the interactions
test
253
Inc.
126
2/10/2011
Lab 3: Conclusion
Parallel delays may seem counterintuitive at first
It is a real effect and does happen in the real world
This example
p is exaggerated
gg
to make it easyy to visualize
test
254
Agenda
Overview of TCP
- TCP Basics
- TCP Windowing
- Lab 4: Examining TCP Behavior
Troubleshooting
T
bl h ti TCP
- Multi-User QuickPredict
- Bandwidth-Delay Product
- Lab 5: Troubleshooting a Slow File Transfer
- Detecting Packet Loss
- Frozen Window
- Nagle
Nagless Algorithm
test
Discrete Event Simulation
Instructor Demo: Nagles Algorithm using DES
References
- Whats New ?
- Wrap up
255
Inc.
127
2/10/2011
TCP Basics
test
What is TCP?
Transmission Control Protocol
Transport
p layer
y protocol
p
Application
Application
Layered between IP and application layer

TCP
Typical applications such as

HTTP (Web)
POP, IMAP, SMTP (e-mail)
Telnet, FTP, and more
test
257
IP
Data Link
Inc.
128
2/10/2011
Why Does TCP Exist?

1) TCP offers reliable data transport to applications
User data is guaranteed to be delivered
2) TCP makes applications behave in a network-friendly manner
TCP may throttle the rate at which it sends user data

Both impact application performance
test
258
Which Applications Dont Use TCP?

TCP adds overhead
TCP limits throughput
Not all applications
pp
need what TCP provides
p
VoIP Voice over IP
Video Conferencing
SNMP Simple Network Management Protocol
Network Gaming Protocols
Domain Name System (DNS)
Network Time Protocol (NTP)
test
259
Inc.
129
2/10/2011
Motivation Why Is TCP Important?

Example TCP/IP Stack
Application Developers
Network Managers
SOAP
SOAP
HTTP
HTTP
SSL
SSL
TCP
TCP
IP
IP
IP
ETHERNET
FIBER
ETHERNET
test
TCP is in a key location of networked applications protocol stacks

Frequently misunderstood
This makes people and software that understand TCP valuable
260
Review: ACE Application Message Chart
Timeline
Tiers
Application
message
arrows
Application
message
groups
g
p
test
Packet
sizes
Red packets have no payload (usually acknowledgements, SYNs, or FINs)

261
Inc.
130
2/10/2011
How to Read TCP Decodes

Special flags
Size is size of entire packet

(including IP, MAC, etc.)
Sequence
number
Receive window
Destination port
Frame number
F
b ttaken
k ffrom
trace file(s). Maytest
not always
be in strict order due to trace
merge
Source port
Payload length
Acknowledgement
number
262
Opening a Connection
Three-way handshake to open connection
Used to exchange
Initial SEQ numbers
TCP Options
Cli
Client
3
C
1
A
test
2
B
Server
263
Inc.
131
2/10/2011
Closing a Connection
Four-way handshake to close connection
I have no more data to send.
Typical example
connection closed at client
client
273
B
274
C
test
275
D
272
A
connection closed at server
264
server
Closing a Connection (cont.)

Often only three segments used
Second FIN also is an acknowledgement for first FIN (B&C in one packet)
TCP Resets are an
alternate way to
terminate a connection
86
A
88
D
87
B&C
test
265
Inc.
132
2/10/2011
SEQ and ACK Numbers

Ensures that data was received
Sending TCP gives each segment a sequence number
SEQ of segment K is SEQ of segment K-1 plus length of segment K-1
Receiving TCP acknowledges reception by replying with ACK

ACK number is the SEQ number of the next expected byte
SEQ and ACK numbers are in

bytes, not packets
Receiver
Sender
SEQ=100
Q
LEN=10
K-1
K
test
ACK= 110
?
Seq # K-1
266
Seq # K
SEQ = 110
? LEN=55
SEQ and ACK Quiz

Which frame(s) is 36 ACKing?
Which frame(s) is 44 ACKing?
test
267
Inc.
133
2/10/2011
TCP Acknowledgements
Receiver normally ACKs every other packet
Inefficient to ACK every packet
What if only one packet is received?
Data
test
268
Acknowledgeement
Delayed-ACK timer must expire

100-200 ms (OS determines exact delay)
Usually does not impact total application performance
Review: TCP Basics

TCP is a connection-based protocol
It uses a three-way handshake for opening connections
Each side closes the connection separately
p
y
SEQ and ACK numbers ensure reliable delivery
test
269
Inc.
134
2/10/2011
TCP Windowing
test
TCP Windowing
TCP window size is the maximum amount of data TCP will allow
outstanding (in-flight)
If window size is less than or equal to outstanding data, TCP will not send
data
In trace below, server is waiting for ACKs before sending more data
Client
test
Server
271
Inc.
135
2/10/2011
TCP Windowing
TCP uses the minimum size of the three windows
Congestion control window
Changes during connection
Internal to sending TCP
Not visible in real world
Lives in the mind of the sender
Receive buffer
May change during connection
User-configurable
Visible as WIN= in decodes
Send buffer
test
Controlled by sending
application
Not visible in decodes
Rarely the bottleneck
272
Congestion Control Window

Receive Buffer
In-Flight Data
TCP Congestion Control Window and

Slow Start
Estimate of how much data the network can handle
Sendingg TCP increases the congestion
g
control window each time it receives
an ACK
Congestion control window increases exponentially every round-trip time
Actually stored in bytes: 1460, 2920, 5840, 11680

When TCP first starts transmitting data, it goes slowly
Sender starts by sending one segment, then waits for ACK
test
273
Inc.
136
2/10/2011
TCP Congestion Control Window in Slow Start

Sender
Receiver
CWND = 1
Data (S1)
ACK (S2)
CWND = 2
Data (S2)
Data (S3)
ACK (S4)
CWND = 3
Data (S4)
Data (S5)
Data (S6)
CWND = 4
test
CWND = 7
274
Data (S7)
Data (S8)
ACK
(S8)
Data (S9)
Data (S10)
Data (S10)
(S11)
ACK
Data (S10)
ACK (S12)
Bulk Data Transfer

Congestion window is greater than receive window
TCP sends data as soon as it receives acknowledgements
In-flight
g data is constant
In-flight data = outstanding data = unacknowledged bytes
Data transfer limited by receive window
test
When you see a pattern like this, blame TCP

275
Inc.
137
2/10/2011
Review: TCP Windowing

TCP limits the amount of outstanding data
Minimum of three windows is used
Send window: How much data the sender can handle
Receive window: How much data the receiver can handle
Congestion control window: How much data the network can handle
Congestion window
Initially small (1, 2, or 4 segments)
Grows exponentially during slow start
test
276
LAB 4
LAB 4
Examining
TCP Behavior
test
277
Inc.
138
2/10/2011
Lab 4: Examining TCP Behavior

Use AppTransaction Xpert to examine TCP behavior
You will see
Connection opens
Slow start
Congestion control window
Raise your hand, and a TA will assist you with any questions
test
278
Lab 4: Conclusion
TCP uses SEQ and ACK numbers to ensure reliable delivery
Minimum of three windows (congestion, receive, and send) determines
whether TCP sends more data
Comparison of bulk and chatty applications
Window size and bandwidth determine throughput for bulk applications
Latency determines throughput for chatty applications
test
279
Inc.
139
2/10/2011
When to Suspect TCP as a Bottleneck

You should suspect TCP of causing problems when:
You cannot explain why an application is running slowly
The servers and network are adequately provisioned
Servers have low load
Sufficient bandwidth
Links not congested
The ACE Live Response Time Composition Chart shows
Large Payload Transfer (Xfer) Times
Large Retransmission Delays
AppTransaction Xpert s AppDoctor Summary of Delays tab shows
Large Network Transfer Effects
test
Large Protocol delays
AppTransaction Xpert s AppDoctor Diagnoses point to
One or more TCP-related bottlenecks or potential bottlenecks
280
Troubleshooting TCP with ACE Live

- Excellent for focusing troubleshooting, monitoring TCP health
Where to start looking, where and when to capture
Measures connection resets, packet loss, long response times, and more
ACE Live Response Time Composition Chart

Large Retransmission Delay
May be sign of packet loss
May also be unfiltered duplicate packets
Large Payload Transfer Times
May be TCP-related
May also indicate bandwidth, congestion or even server processing causes
test
281
Inc.
140
2/10/2011
Troubleshooting TCP with AppTransaction Xpert

AppDoctor provides quick way to check for TCP problems
Suspect TCP if the Summary of Delays shows
Large Network Transfer Effects
Large Protocol delays
Use diagnosis tab to identify which

specific TCP-related issue affects
your trace
Bottlenecks
Potential bottlenecks
test
282
Understanding Network Transfer Effects

Combination of three effects related to transferring data
Bandwidth, Congestion, Protocol
Bottleneck bandwidth is needed to separate these phenomena

Refine Network Effects key to identification of delays due to the TCP protocol
test
283
Inc.
141
2/10/2011
How to Identify TCP Problems

Look at the AppDoctor Diagnosis
tab
Effect of Protocol
Effect of Network Transfer
Last 7 stats deal with TCP
Helps target the specific
TCP issue
Look at decodes for long

application transactions
Look in the network
test and
chart at the start
end of long delays
284
How to Identify TCP Problems (cont.)

Double-click in the Summary of Delays for more information
Use separate drilldown screens for Protocol and Congestion
test
285
Inc.
142
2/10/2011
How to Identify TCP Problems (cont.)

Congested networks also can cause protocol delays
TCP implements congestion avoidance
Single-sided captures of receiving data may make congestion look like
protocol delay or tier processing
test
286
Review: Analyzing TCP Behavior

Look for TCP problems in traces with large amount of protocol delay
Congestion delays also induce protocol delays
AppDoctor
pp
(forensic)
(
)
Separately diagnoses protocol and congestion
Diagnoses several TCP problems
QuickPredict (what if?)

Separately models protocol and congestion
Can simulate changing TCP Window Size
test
287
Inc.
143
2/10/2011
Review: ACE QuickPredict

Quick simulation engine
Combination of analytical and discrete event simulation
Uses analytical models of TCP behavior
Initially shows a pristine view of network

Allows you to study effects of the network on the application
Outputs graphs of response time vs. one of the following:
Bandwidth
Latency
Packet loss
Link utilization
TCP receive window size
NEW: WAN Acceleration parameters
QuickPredicttest
Bar Charts supports
asymmetric window sizes
288
Review: QuickPredict Input and Output

Y axis is always response time
X axis is a range of values for one of:
Bandwidth, Latency, Packet Loss, Link Utilization, TCP Window size,
and WAN Acceleration (NEW!)
Change values of other variables

to see impact on response time
test
289
Inc.
144
2/10/2011
Multi-User QuickPredict
test
Multi-User QuickPredict (MUQP)

Single-user QuickPredict considers one client only
MUQP allows you to have more than one
Client and client site
Applications (either ACE traces or ACE Whiteboard traces)
Data centers
Output
Response time (including
delay components)
Utilization
Results
Average test
N-th percentile
291
Inc.
145
2/10/2011
Inputs to MUQP
Deployed applications (defined in ACE and/or ACE Whiteboard files)
Network topology (number of client sites, clouds, and data centers)
Tier-pair
p pparameters ((bandwidth, latency,
y link utilization, window size, and
packet loss)
Application deployment options (number of transactions per hour for each

client site)
test
292
Outputs of MUQP
"Average-case" response times for applications
"Worst-case" results for access links (utilization) and applications (total
response time and components of tier processing, latency, protocol, and
congestion delay)
test
293
Inc.
146
2/10/2011
Instructor Demo of MUQP
test
294
Bandwidth-Delay Product
test
Inc.
147
2/10/2011
Bandwidth-delay product Defined as amount of data (in bytes) that can be
sent before ACK is received for the first unacknowledged bit of data
2 x one-way latency x bandwidth of slowest link
To fill the pipe, TCP window size must be equal to bandwidth-delay product
If window is too small, TCP must wait for acknowledgment
g
before sendingg new data
Example below
Bandwidth:
Latency:
BWDP:
Receive Window:
Throughput limited to:
1,544 kbps = 193 KB/sec

250 ms = 0.250 sec
2 x .25 x 193 = 96.5 KB
8 KB (1/12 ideal size)
130 kbps (1/12 link speed)
test
296
Capacity = .096 s 125 KB/s = 12 KB (8 segments)
Bandwidth Delay Product = 2 Capacity = 24 KB (16 segments)
Bandwidth-Delay
96 ms
Sender
Receiver
test
1 Mbps
297
Inc.
148
2/10/2011
LAB 5
LAB 5
Troubleshooting
a Slow File Transfer
test
298
Lab 5: Troubleshooting a Slow File Transfer

Overview
Computer manufacturer moving bulk data from Chicago to Taipei has dedicated
T1 without any packet drops
Not seeing expected response time
You are a network engineer
Your manager: It must be the network
Contemplating (expensive) link upgrade
Objective
Determine the cause of slow
response time
Propose and test a solution
test
299
Inc.
149
2/10/2011
Lab 5: Conclusion
Simple TCP configuration change improved performance
Over same time period:
p
Large receive window
Small receive window

test
What is the downside of making this change?

300
Changing the Receive Window

How do I change the receive window on my computer?
Windows
http://www.microsoft.com/technet/network/tcpip/default.mspx
http://www.microsoft.com/technet/itsolutions/network/deploy/depovg/tcpip
2k.mspx
Google search: windows tcp/ip implementation details
Solaris
http://www.sean.de/Solaris/soltune.html#water
Linux
http://www.speedguide.net/read_articles.php?id=121
Default for modern operating systems is 64 KB

test
Maximum advertised receive window value in TCP packet is 64 KB

How do we handle todays large capacity, high latency pipes?
TCP Window Scaling
301
Inc.
150
2/10/2011
TCP Window Scaling

TCP allows for WIN= field to be scaled by a power of 2
Decodes will show WIN<<2=262140
WIN<<x means multiply field by 2x
This setting is exchanged in TCP options at beginning of connection

Both sides must agree and understand this option
test
What if a packet trace does not contain the opening of a connection?

AppTransaction Xpert cannot detect the presence of TCP Window scaling
302
Detecting Packet Loss
test
Inc.
151
2/10/2011
TCP and Packet Drops

Where do packets get dropped, and why?
Media collisions
Wireless signal interference
Congested networks
Endpoints must react to overloaded routers
Drops are a form of indirect communication between routers and senders
Routers have multiple strategies for keeping queue size in check
Tail drop
Random Early Detection (RED)
test
304
Packet Loss and Retransmissions

Packet is dropped by the network and retransmitted
Time between when it should have arrived and when it did arrive
Controlled by TCP timers
Counted
C
t d as P
Protocol
t
lD
Delay
l
Retransmission also invokes congestion control
Slows down throughput
Counted as Protocol Delay T = 0 1 2 3 4 5 6 7 8 9 10
Client
test
Protocol Delay
Server
0 1 2 3 4 5 6 7 8 9 10
305
Inc.
152
2/10/2011
Drops and Retransmissions

TCP must make sure all bytes of a stream are received
TCP guarantees delivery by retransmitting packets presumed dropped
Two methods by which TCP will retransmit a dropped packet

Retransmission timeout
Sending TCP schedules a timer for outstanding packets
Timer is cancelled when the packet is acknowledged
If timer expires, packet is presumed missing, re-sent
Triple duplicate ACKs

ACK number indicates next expected packet
If receiving TCP does not get the expected packet, it ACKs every packet
test
If receiving TCP does not get the expected packet, ACK number does not
increase.
If sending TCP receives acknowledgements with the same ACK number, it
knows a packet was likely lost
306
TCP One Loss: Retransmission Timeout

Sender
Receiver
Data (S101)
Data (S102)
Data (S103)
Data (S104)
ACK ((S103))
Data (S105)
ACK (S105)
Data (S106)
Data (S107)
Data (S108)
Retransmission
timeout
ACK (S106)
ACK (S106)
test
Data (S106)
ACK (S109)
307
Inc.
153
2/10/2011
TCP One Loss: Triple Duplicate ACK

Sender
Receiver
Data (S101)
Data (S102)
Data (S103)
Data (S104)
ACK (S103)
Data (S105)
Triple duplicate
ACK and fast
retransmit
Data (S106)
ACK (S104)
Data (S107)
ACK (S104)
Data (S108)
ACK ((S104))
test
Because 104 was

not received,
Receiver does not
change ACK
number
ACK (S104)
Data (S104)
ACK (S109)
308
Cost of Loss Detection

Retransmission timeout
There are not enough ACKs for triple duplicate ACK
Cost: Timer (often large) plus time to send 1 packet
Congestion window decreases to one segment
Triple-duplicate ACKs
Cause fast retransmit
Cost: Time to send 1 packet
Retransmission
Triple-duplicate ACK
test
Notice the send times
309
Retransmission
Inc.
154
2/10/2011
Review: Detecting Packet Loss

Two mechanisms
Retransmission time-out (RTO)
Costly
Always works
Triple-duplicate ACK
Relatively efficient
Does not always happen
Chatty applications rarely qualify
Food for thought

Why acknowledge every packet during loss?
Why not quintuple-duplicate
ACK?
test
Why not single-duplicate ACK?
310
Frozen Window
test
Inc.
155
2/10/2011
TCP Frozen Window

Receiver stores packets in receive buffer
Advertised receive window WIN= reduced by data in buffer
Normal applications
pp
remove data quickly
q
y
What happens if:
A server is handling too many clients?
What if the server is running other applications?
What if a another thread is hogging the CPU?
test
312
TCP Frozen Window (cont.)

As WIN decreases, buffer fills up, sender cannot send data
Connection becomes frozen
Sender will send pprobe packets
p
to see if window size has increased
test
313
Inc.
156
2/10/2011
TCP (Almost) Frozen Windows

As WIN decreases, sender cannot send data
What if the WIN is less than MSS?
Sender could send data, but often does not
Detection of chilly windows new to AppTransaction Xpert 16.0
test
314
Drilling into Protocol (cont.)
test
Frozen Windows:
Receiving WIN goes to zero
Sending side sends probe packets
315
Inc.
157
2/10/2011
Nagles Algorithm
test
Nagles Algorithm
Nagles Algorithm
TCP mechanism to reduce sending of many tiny messages
Sender may not send data unless a full, MSS-sized segment can be sent
OR there is no unacknowledged data
Delayed Acknowledgement
Interaction of these two mechanisms sometimes causes problems
Ridiculous situation where both sides wait for the other to do something
Client
test
B
Server
317
200 ms
Inc.
158
2/10/2011
Nagles Algorithm Example

Client sends segment A (less than MSS)
Client wants to send segment B, but needs to wait for ACK
Server is waitingg for another segment
g
before sendingg ACK
Eventually, server sends ACK

Client then sends segment B
Process repeats
Client
test
B
Server
318
200 ms
Nagles Algorithm Example (cont.)

What causes the delay for this application?
test
319
Inc.
159
2/10/2011
Nagles Algorithm and Delayed ACK

To eliminate delays when sending consecutive small messages:
Rewrite the application
Sending multiple small consecutive messages is inefficient
Sometimes you cant change the application
Disable Nagles algorithm
Done within the application
Effects only the application changed: (TCP_NODELAY socket option)
Not always possible to change an existing application
Acknowledge every packet
Practical real world alternative
On Windows: http://smallvoid.com/article/winnt-nagle-algorithm.html
test
Warning: Applies to every application on the machine!
Interacts with congestion control window, affecting bulk applications
320
References
TCP in OPNET
Online documentation
Overview PDF
Model Library/Standard / TCP Model Usage Guide
ACE TCP Windowing
Wi d i Bottleneck
B ttl
k online
li documentation
d
t ti
Applications in OPNET
Several whitepapers including SMARTE
Help / Product Documentation / Modeling Methodologies / Methodologies and Case
Studies
http://www.psc.edu/networking/projects/tcptune/
Setting system wide TCP window sizes
Listing of TCP features for each operating system
http://www.ncsa.uiuc.edu/People/vwelch/net_perf/tcp_windows.html
Instructions on setting TCP window sizes within application code
http://www.sean.de/Solaris/soltune.html
test
Tuning your TCP/IP stack on Solaris
TCP/IP Illustrated, Volume I, Richard Stevens
321
Inc.
160
2/10/2011
Take-Away Points
Windowing causes TCP to wait before transmitting application data
Bulk data-transfer throughput often limited by TCP windowing
TCP frequently
q
y does not use the full available bandwidth
Packet loss can cause significant reduction in throughput
QuickPredict is a fast, easy, accurate way to predict application response time
Use DES to model and study detailed behavior
Simulate using the proper TCP machine settings
Next step: Run homework labs for a greater understanding of TCP
test
322
Whats New ??
test
Inc.
161
2/10/2011
Whats New in AppTransaction Xpert ??

64-Bit Support
Access more that 4GB of memory on 64-bit machines which allows you analyze
much larger capture files
Windows 7 Professional (32-bit and 64-bit)

Capture Enhancements
Manage captures from ACE Live appliances within the Application Capture
Manager. New ACE Live tab to quickly download and preview data
ACE Capture Agent 3.9 Ability to filter the traffic data by time range, IP
addresses, and/or IP conversations.
Capture File Filters Greater flexibility in specifying hosts and ports.
Support for Riverbed Transparent Mode

test
The AppTransaction Xpert import now supports capture files from Riverbed
accelerators running in full or port transparency mode
324
Whats New in AppTransaction Xpert (cont.)

Streamlined Import Process
Quickly open capture files through the user interface without specifying any information
(such as location, bandwidth, latency, etc.)
Bandwidth,
Bandwidth Latency
Latency, and Locations can be specified as needed
Automatic Filtering of Duplicate Packets and TSO (TCP Segmentation Offloading)
Simplified Manual Merge Workflow
QuickPredict Enhancements - WAN Acceleration Support

Support for WAN optimization technologies by allowing users to predict behavior for
compression, caching, and application acceleration
HTTP Analysis
Anal sis Enhancements
test
Automatic detection
of HTTP on non-standard ports (thus eliminating the need to perform
redecodes or modifying config files)
Improved analysis for packet slicing and missing packets
HTTP-specific visualization of object response times in TreeView
325
Inc.
162
2/10/2011
References
test
Additional Documentation
Online at:
www.OPNET.com / Support / Support Center
Documentation
Release Notes
Also look at: http://userforums.opnet.com

Info on ALL OPNET products
Discussion Groups
Hints
Hi & tricks
i k
test
327
Inc.
163
2/10/2011
OPNET Training Courses

OPNET provides Instructor-led training for all of our
products.
OPNET
OPNET Training Facilities;
Bethesda, Cary, Dallas, Nashua, Santa Clara
On-site at YOUR facilities

On the Web via WebEx
For AppTransaction Xpert :

2-day Instructor-led-training
test
Register for
courses online at:
www.opnet.com/Training
328
OPNET Certification Program

Become recognized as an expert user of OPNET products
Certification covers relevant topics such as:
Management / Administration
Configuration
Product Selection
Product Operation
Networking Knowledge
Upon
p successful completion:
p
Letter
test
Certificate
Individual certification ID
Valid for 2 years
329
Inc.
164
2/10/2011
AppTransaction Xpert Online Training
test
330
OPNET Web Pages - Training Resource
test
331
Inc.
165
2/10/2011
OPNETWORK is the largest event of its kind,

attracting thought leaders from industry, government,
and academic communities from all over the world
world, to
advance best practices
for:
test
Application Performance Management
Network Engineering, Operations, and Planning
Network R&D
RSVP at: www.opnet.com/opnetwork2010

332
332
Course Evaluation
Well use this information to improve our courses and teaching methods. Please
enter as much comment material as youd like the more info you add, the
better well get!
Point your browser to:
http://eval.opnet.com
Please enter Instructor and Course info
Course: Introduction to AppTransaction Xpert
Course Date:
Instructor Name:
FREE
Stuff !!
test
Thank You in advance for your participation!
333
Inc.
166
2/10/2011
test
334
Inc.
167

AppTransaction Xpert

Hochgeladen von

Dokumentinformationen

Originalbeschreibung:

Copyright

Verfügbare Formate

Dieses Dokument teilen

Dokument teilen oder einbetten

Freigabeoptionen

Stufen Sie dieses Dokument als nützlich ein?

Sind diese Inhalte unangemessen?

Copyright:

Verfügbare Formate

AppTransaction Xpert

Hochgeladen von

Copyright:

Verfügbare Formate

Lecture Manual

Introduction to ACE Analyst

Destination Control Statement

Restricted Rights Legend

OPNET Technologies, Inc. (OPNET) is the sole owner

USE BY AGENCIES OF THE UNITED STATES

Introduction to ACE Analyst

Destination Control Statement

Restricted Rights Legend

OPNET Technologies, Inc. (OPNET) is the sole owner

USE BY AGENCIES OF THE UNITED STATES

About OPNET Technologies, Inc.

Customers (partial list)

Healthcare / Bio / Pharmaceutical

OPNETs APM Xpert Suite

Deep application component

Deep transaction tracing,

Network and server

What's in your Application?

OPNET Confidential Not for release to third parties

OPNET is an IACET Authorized Provider

Continuing Education Units

Earning CEUs for this Course

To receive these CEUs, you must

Attend the entire course

Your test is scored in the next 10 business days

Lets get to know each other

Class is interactive dont hesitate to ask questions

Using AppTransaction Xpert to visualize applications

Using AppTransaction Xpert to diagnose problematic applications

Reporting capabilities of AppTransaction Xpert

Using AppTransaction Xpert to perform performance predictive studies

AppTransaction Xpert Solution

ImportantExisting users are not losing any AppTransaction Xpert functionality

What is AppTransaction Xpert... ?

Diagnose performance problems

Predict application behavior under different network conditions

When to use AppTransaction

Users of a critical application are reporting very bad response times

Result: Finger pointing !!

Relocation of key processing data center from US to China

Result: Need to produce accurate performance projections and network

Purchase of a new 3-party application system

What does AppTransaction Xpert tell you ?

Users desktop running slow?

Can this new application be deployed over our network?

How does AppTransaction Xpert tell you this...?

Inputs to AppTransaction Xpert

Basic characteristics of network (Recommended)

Application Troubleshooting Workflow

Optional: Create IT Guru

AppTransaction Xpert Visualizations

Tier Pair Circle

View time-based graphs of application statistics

Data Exchange Chart (DEC)

Slope indicates time to cross

Color indicates payload size

DEC: Message Groups

DEC: View Options

DEC: Show Dependencies

Click Show Dependencies

White = delay at host

DEC: Application Characteristics