Lecture Manual

Transcription

Lecture Manual

Lecture Manual
Introduction to ACE Analyst
NOTICES
Product Information
Lecture Manual for: ACE Analyst
Software Release: 16.0.T PL0
Destination Control Statement
This Lab Manual and the OPNET Software were
manufactured in the United States by OPNET. The initial
export of this Lab Manual and the OPNET Software, and
any subsequent relocation to another country shall comply
with United States law relating to the export of technical
data, equipment, software and know-how.
Technical Support
Restricted Rights Legend
Telephone:
240.497.1200
Facsimile:
240.497.1064
E-mail:
support@opnet.com
Web:
http://www.opnet.com
OPNET Technologies, Inc. (“OPNET”) is the sole owner
of all proprietary rights in this Lab Manual (the “Lab
Manual”) and the OPNET IT Guru software (the
“OPNET Software”).
Terms and Conditions of Use
Access to and use of the Training Manual and/or the
OPNET Software is restricted to authorized users who
are employees or direct contractors of a Licensee who
has accepted the terms and conditions of OPNET’s
Software Master License Agreement. As a condition of
having access to or use of the Lab Manual and/or the
OPNET Software all users agree to be subject the terms
of OPNET’s Software Master License Agreement. Users
shall not copy or reproduce, in whole or in part, or
permit access to or disclose the Lab Manual or the
OPNET Software to any other person without the
prior written consent of OPNET.
Liability Statement
The terms and conditions governing the licensing of
OPNET Software consist of those set forth in the written
contracts and license agreements between OPNET and its
customers. No representations in this Lab Manual or
statements of capability, suitability for use, accuracy or
performance, shall be a warranty by OPNET unless
contained in a written agreement signed by OPNET.
OPNET DISCLAIMS ALL EXPRESS OR IMPLIED
WARRANTIES AS TO THE LAB MANUAL AND
THE OPNET SOFTWARE INCLUDING IMPLIED
WARRANTIES OF MERCHANTABILITY AND
FITNESS FOR A PARTICULAR PURPOSE.
USE BY AGENCIES OF THE UNITED STATES
GOVERNMENT: For purposes of use of this Lab
Manual and OPNET Software by agencies of the United
States government (or those authorized to use of this Lab
Manual and OPNET Software on behalf of an agency),
the OPNET Software is “Commercial Computer
Software” within the meaning of DFARs 227.7202 and
252.227-7014(a)(1) and FARs 52.227-19. Use,
reproduction and disclosure is subject to DFAR and FAR
provisions relating to Commercial Computer Software
and the License Agreement between OPNET and the
agencies using OPNET Software. OPNET software is
provided with a RESTRICTED RIGHTS NOTICE for
government use.
Trademarks
OPNET, OPNET Technologies, Inc., OPNET Modeler, IT
Guru, OPNET Development Kit, NetBiz and IT Guru are
trademarks of OPNET Technologies, Inc. All other
trademarks used in the Lab Manual are the property of their
respective owners and used herein for identification
purposes only.
CONFIDENTIAL INFORMATION
DO NOT DISCLOSE, FORWARD, DISTRIBUTE,
SHARE, OR MAKE COPIES OF THIS DOCUMENT IN
WHOLE OR IN PART. This document contains
confidential information and may contain information that
is proprietary, privileged, and/or exempt from disclosure
under applicable law. This document is intended for the
exclusive use of the person to whom it is disclosed. If you
are an unauthorized person, you are hereby notified that
any viewing, copying, disclosure or distribution of this
information may be subject to legal action. All
unauthorized persons must immediately destroy the original
documentation without making any copies or further
unauthorized disclosure.
Lecture Manual
Introduction to ACE Analyst
NOTICES
Product Information
Lecture Manual for: ACE Analyst
Software Release: 16.0.T PL0
Destination Control Statement
This Lab Manual and the OPNET Software were
manufactured in the United States by OPNET. The initial
export of this Lab Manual and the OPNET Software, and
any subsequent relocation to another country shall comply
with United States law relating to the export of technical
data, equipment, software and know-how.
Technical Support
Restricted Rights Legend
Telephone:
240.497.1200
Facsimile:
240.497.1064
E-mail:
support@opnet.com
Web:
http://www.opnet.com
OPNET Technologies, Inc. (“OPNET”) is the sole owner
of all proprietary rights in this Lab Manual (the “Lab
Manual”) and the OPNET IT Guru software (the
“OPNET Software”).
Terms and Conditions of Use
Access to and use of the Training Manual and/or the
OPNET Software is restricted to authorized users who
are employees or direct contractors of a Licensee who
has accepted the terms and conditions of OPNET’s
Software Master License Agreement. As a condition of
having access to or use of the Lab Manual and/or the
OPNET Software all users agree to be subject the terms
of OPNET’s Software Master License Agreement. Users
shall not copy or reproduce, in whole or in part, or
permit access to or disclose the Lab Manual or the
OPNET Software to any other person without the
prior written consent of OPNET.
Liability Statement
The terms and conditions governing the licensing of
OPNET Software consist of those set forth in the written
contracts and license agreements between OPNET and its
customers. No representations in this Lab Manual or
statements of capability, suitability for use, accuracy or
performance, shall be a warranty by OPNET unless
contained in a written agreement signed by OPNET.
OPNET DISCLAIMS ALL EXPRESS OR IMPLIED
WARRANTIES AS TO THE LAB MANUAL AND
THE OPNET SOFTWARE INCLUDING IMPLIED
WARRANTIES OF MERCHANTABILITY AND
FITNESS FOR A PARTICULAR PURPOSE.
USE BY AGENCIES OF THE UNITED STATES
GOVERNMENT: For purposes of use of this Lab
Manual and OPNET Software by agencies of the United
States government (or those authorized to use of this Lab
Manual and OPNET Software on behalf of an agency),
the OPNET Software is “Commercial Computer
Software” within the meaning of DFARs 227.7202 and
252.227-7014(a)(1) and FARs 52.227-19. Use,
reproduction and disclosure is subject to DFAR and FAR
provisions relating to Commercial Computer Software
and the License Agreement between OPNET and the
agencies using OPNET Software. OPNET software is
provided with a RESTRICTED RIGHTS NOTICE for
government use.
Trademarks
OPNET, OPNET Technologies, Inc., OPNET Modeler, IT
Guru, OPNET Development Kit, NetBiz and IT Guru are
trademarks of OPNET Technologies, Inc. All other
trademarks used in the Lab Manual are the property of their
respective owners and used herein for identification
purposes only.
CONFIDENTIAL INFORMATION
DO NOT DISCLOSE, FORWARD, DISTRIBUTE,
SHARE, OR MAKE COPIES OF THIS DOCUMENT IN
WHOLE OR IN PART. This document contains
confidential information and may contain information that
is proprietary, privileged, and/or exempt from disclosure
under applicable law. This document is intended for the
exclusive use of the person to whom it is disclosed. If you
are an unauthorized person, you are hereby notified that
any viewing, copying, disclosure or distribution of this
information may be subject to legal action. All
unauthorized persons must immediately destroy the original
documentation without making any copies or further
unauthorized disclosure.
2/10/2011
Introduction to AppTransaction
Xpert (ACE Analyst)
test
Day 1
OPNET Confidential – Not for release to third parties © 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc.
About OPNET Technologies, Inc.®
Corporate Overview
• Founded in 1986
• Publicly traded (NASDAQ: OPNT)
• HQ in Bethesda, MD
• Approximately 600 employees
• Worldwide presence through direct offices and channel
partners
• Global distribution agreement with Cisco
Best-in-Class Solutions and Services
• Application Performance Management
• Network Performance Management
• Network R&D
Strong Financial Track Record
g historyy of profitability
p
y
• Long
• Trailing 12-month revenue of over $100M
• Approximatelytest
25% of revenue re-invested in R&D
Broad Customer Base
• Corporate Enterprises
• Government Agencies/DoD
• Service Providers
• Network Equipment Manufacturers
3
OPNET Confidential – Not for release to third parties © 2010 OPNET Technologies, Inc. All
rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies,
Inc.
1
2/10/2011
Customers (partial list)
Financial Services
Government
Insurance
Healthcare / Bio / Pharmaceutical
Manufacturing
Energy / Utilities
Media / Communications
Services
test
Retail / Consumer Products
4
Transportation / Logistics
OPNET’s APM Xpert Suite
End-user experience and network
monitoring and analytics
Deep application component
monitoring and analytics
Deep transaction tracing,
analysis, and prediction
Passive database
performance monitoring
test
Network and server
discovery and monitoring
“What's in your Application?”
5
OPNET Confidential – Not for release to third parties
© 2011 OPNET Technologies, Inc. All rights reserved. OPNET and OPNET product names are trademarks of OPNET Technologies, Inc.
Inc.
2
2/10/2011
OPNET is an IACET Authorized Provider
OPNET Technologies, Inc. has been approved as an Authorized Provider by the
International Association for Continuing Education and Training (IACET), 1760 Old
Meadow Road, Suite 500, McLean, VA 22102. In obtaining this approval, OPNET
has demonstrated that it complies with the ANSI/IACET 11-2007
2007 Standard which is
internationally recognized as the Standard for training organizations.
As a result of their Authorized Provider status OPNET is authorized to offer IACET
Continuing Education Units (CEUs) for its programs that qualify under the
ANSI/IACET 1-2007 Standard.
test
6
Continuing Education Units
What is IACET?
IACET is a Standards Developing Organization accredited by the American Standards Institute
(ANSI) and is an authorizing body for education providers that meet the American National
Standard for continuing education and training (ANSI/IACET 1-2007 Standard). The IACET
standard is what learners seek for quality and is the core of thousands of educational programs
worldwide. (see: www.IACET.org)
What is an Authorized Provider?
The IACET Authorized Provider (AP) status means that an organization adheres to IACET’s
internationally recognized ANSI/IACET 1-2007 Standard for Continuing Education and Training.
Only Authorized Providers may award IACET Continuing Education Units (CEUs).
Whatt is
Wh
i a CEU?
test
The Continuing Education
Unit (CEU) was created by IACET as a measurement of continuing
education. One (1) IACET CEU is equal to ten (10) contact hours of participation in an organized
continuing education experience under responsible sponsorship, capable direction, and qualified
instruction.
7
Inc.
3
2/10/2011
Earning CEUs for this Course
Introduction to AppTransaction Xpert
– This course is worth 1.3 CEUs
• To receive these CEUs, you must
–
–
–
Attend the entire course
Fill out the feedback form
Score 80% in a test at the end of class
• Your test is scored in the next 10 business days
• If successful, you will receive a certificate denoting your CEUs in the mail
within 2 weeks
test of the course is stored in our database for later retrieval as
• Your completion
needed
8
Let’s get to know each other…
• Name ?
• What company you work for ?
• Which Department ?
• What experience do you have with OPNET products?
• Did you know we changed the names… !!??
test
9
Inc.
4
2/10/2011
Course Structure
10
Day 1
Introduction to AppTransaction Xpert
Day 2
Advanced AppTransaction Xpert
•About OPNET
•Visualizations in AppTransaction Xpert
•Troubleshooting application
performance
•Predicting application behavior before
deployment
test strategies
•Capture and import
•WAN Acceleration
•Advanced Topics
•Understanding Delay Components
•Parallel Effects, User think Time
•Troubleshooting TCP applications
Class Format
• In general, four phase instruction
– Short lecture sections
– Demonstrations
– Related hands-on labs
– Discussion of lab applications
• Class is interactive – don’t hesitate to ask questions
• Please fill out the feedback form online @ eval.opnet.com
– Give us information on your current network and application concerns
test
– Provide suggestions on how we can improve our products
– Let us know how we can improve this training class
11
Inc.
5
2/10/2011
Agenda
 Introduction
- Licensing Options
- Overview, Use Cases
- Basic Workflow
 Using AppTransaction Xpert to visualize applications
- AppTransaction Xpert Visualizations
- Lab 1: Visualizing Application
 Using AppTransaction Xpert to diagnose problematic applications
- AppDoctor
 Reporting capabilities of AppTransaction Xpert
- Reports
R
t
test
- Lab 2: Application
Performance Troubleshooting
 Using AppTransaction Xpert to perform performance predictive studies
- QuickPredict and QuickRecode
- Lab 3: Predictive Analysis
12
Licensing Options
• Download and administer licenses using the License Manager
• Licensed on concurrent user basis
• Online license management
g
(http://www.opnet.com/support/license_management.html)
• Licensing options
– Standalone vs. Floating (license server)
– Loanable Licenses
test
13
Inc.
6
2/10/2011
Overview of AppTransaction
Xpert
test
AppTransaction Xpert Solution
• AppTransaction Xpert Plus
• AppTransaction Xpert Standard
• IT Guru with the following modules
– ACE
– ACE Decode Module
– Advanced ACE Capabilities
• Starting with version 16.0, the different offerings of AppTransaction Xpert (that is:
AppTransaction Xpert Plus, AppTransaction Xpert Standard, and AppTransaction
Xpert Module) have been combined into a single installer.
test
Important—Existing users are not losing any AppTransaction Xpert functionality
because of this change. Functionality is determined by licenses.
15
Inc.
7
2/10/2011
What is AppTransaction Xpert... ?
• Application Characterization Environment
• Capture application traces
• Visualize
Vi li Application
A li ti Dynamics
D
i
– Application layer analysis
– Network layer analysis
• Diagnose performance problems
– Minimize troubleshooting time
• Validate solutions
test
– Predict application behavior under different network conditions
– Evaluate performance impact of application design decisions
– Validate infrastructure readiness for new deployments
• Generate reports
16
When to use AppTransaction
Xpert ??
(Use Cases)
test
Inc.
8
2/10/2011
Use Case 1
Troubleshooting application performance –
• Users of a critical application are reporting very bad response times
– Network engineers are reporting no issues on the network
– Application engineers are reporting no issues with software
• Result: Finger pointing !!
Wasted time
test
18
Use Case 2
Testing changes of the enterprise environment –
• Relocation of key processing data center from US to China
• Management needs to know the implications of move
– Impact on application performance and network utilization?
– Will additional resources be required?
• Result: Need to produce accurate performance projections and network
planning
l i requirements
i
test
19
Inc.
9
2/10/2011
Use Case 3
Application pre-deployment analysis –
• Purchase of a new 3-party application system
• Management needs to know how the new system will perform
– Application response times
– Network utilization
• Result: Need to eliminate the need for trial and error deployment of
application. Expensive and time consuming.
test
20
Basic Workflow
test
Inc.
10
2/10/2011
What does AppTransaction Xpert tell you … ?
• “Why is my application slow?”
–
–
–
–
–
–
User’s desktop running slow?
Server delays?
y
Non-optimal protocol settings (TCP and others)?
Heavy congestion over WAN?
Too little bandwidth?
Application too chatty?
• “Can this new application be deployed over our network?”
– Expected test
transaction response times?
– Impact other critical applications?
– Bandwidth requirements?
22
How does AppTransaction Xpert tell you this...?
• Reports, graphs, and tables
• Visualization of application
pp
behavior
• Diagnosis of bottlenecks and performance problems
• Predictive analysis of same traffic on different network
• Input to other OPNET products for further analysis
– Application traffic analysis
test
– Capacity planning studies
23
Inc.
11
2/10/2011
Inputs to AppTransaction Xpert
• Packet traces from the following sources
– ACE Capture Agents
– ACE Live
– Cisco NAM Blade
– WAN Accelerator
– F5 BIG-IP
– Sniffer
– UNIX (no agent installed)
– Tcpdump, windump ..etc
• Basic characteristics of network (Recommended)
test
– Tier locations
– Bandwidth between tiers
– Latency between tiers
24
Application Troubleshooting Workflow
Identify performance issues
Capture Packet Traces
Visualize, Diagnose, and Predict
Optional: Detailed
Analysis
test
Optional: Create IT Guru
network model
Generate Reports
25
Inc.
12
2/10/2011
AppTransaction Xpert
Visualizations
test
AppTransaction Xpert Visualizations
• Data Exchange Chart
– Shows timeline of data sent
• Tier Pair Circle
– Shows summary statistics
• Tree View
– Shows Transaction Analysis
• Decodes
– Shows Application layer headers
• Graphs
test
– View time-based graphs of application statistics
27
Inc.
13
2/10/2011
Data Exchange Chart (DEC)
• Timeline of application
messages and network
packets between tiers
• Slope indicates time to cross
network
• Color indicates payload size
of message
test
28
DEC: Message Groups
• When messages and packet
are too frequent to be displayed
individually, a solid block is shown
• Block is a histogram showing the
size of messages in groups
• Zoom in or use tooltip to see more
detail
test
29
Inc.
14
2/10/2011
DEC: View Options
•Application Chart Only
Shows every application message – several packets may comprise a message
• Network Chart Only
Shows every individual packet in the trace file, including TCP acknowledgements
• Application and Network Charts
Shows both charts in the same window
test
Application Messages
Network Packets
30
DEC: Show Dependencies
• Triangles appear at largest
chunks of delay
Click “Show Dependencies”
• White = delay at host
• Orange = user think time
• Purple = network delay
test
Legend
31
Inc.
15
2/10/2011
DEC: Application Characteristics
• Quiet Time
–
–
–
–
User think time?
Slow client computer?
p
Slow server?
Not seeing the whole picture?
§Tip: See what happened before and
f silence
after
test
Tip: Check Network Chart for
more details
Large periods of silence
32
DEC: Application Characteristics (cont.)
• Chatty Application
– Lots of back-and-forth communication
(Application Turns)
Lots of small messages
(orange and yellow)
test
Groups of messages close together
33
Inc.
16
2/10/2011
Application Turn
• Counted on the Application Layer (Application Messages)
• Every turn has to cross the network latency which adds up to the total
Latency Delay
2
Client
Network
Latency
test
Server
1
34
3
DEC: Application Characteristics
• One-way data transfers
– Dark blue arrows
– Most arrows in same direction
Tip: Check the Network Packet
Chart for more details
test
Zoom to see the arrow heads
35
Inc.
17
2/10/2011
DEC: Application Characteristics (cont.)
• Lots of irrelevant
traffic
– Do not try diagnose
application until
you removed
irrelevant data
test
Tip: Go to Tier Pair
Circle and delete
unrelated traffic
36
Tier Pair Circle
• Summary of traffic between all tiers
g
– A high-level
view of the
application task
– Is there extraneous traffic?
• Right-click to exclude
unwanted traffic
test
Tip: If you don’t see the protocol you expect, then right-click to re-recode traffic on the
fly
37
Inc.
18
2/10/2011
Tier Pair Circle: Functionality
• Choose statistics to view
• Right-click a tier or
conversation to exclude
/delete it or all others
• Show only traffic containing
a specific protocol
• Find Top
p Statistics to sort
conversations by
relevance
test
38
Tier Pair Circle: Statistics
• Pull down to select statistics to
view:
test
39
Inc.
19
2/10/2011
Tree View
• Tree pane: One-line summary of tier pairs, connections and frames
• Table pane: Detail statistics by tier pairs, connections and frames
• Timeline ppane: Overview of messages
g over the time of the task
test
Tree Pane
40
Table Pane
Timeline Pane
Tree View: “View by:” Options
• Tier Pairs – Network Packets
• Tier Pairs – Application Transactions
• Application Transactions
Group by:
test
41
Inc.
20
2/10/2011
Tree View: Sorting
• Click a column header to sort the table by descending values.
• Click the table header again to sort by ascending values.
test
42
Tree View : Customize the Table Pane
• Choose View > Choose Table Columns…
• Right-click on the column header and click “Choose TreeView
Columns…”
• Select the network statistics to appear in the table
test
43
Inc.
21
2/10/2011
Decode Viewer
ETH
IP
TCP
HTTP
ETH
• What are decodes?
– Analysis or description of the bytes at each layer of a packet
• What can you test
learn from decodes?
 Detailed protocol information
 Information about data being transferred
 Ex: TCP gives Sequence number and Acknowledgement number
44
Embedded Decode Viewer
• Show / Hide the
embedded decode viewer
by clicking
and
• Select traffic in the top
pane and see
corresponding decodes in
the bottom pane
• Right-click on selected
frames to export to
test
spreadsheet
• Click the plus sign to
expand layers of decodes
45
Inc.
22
2/10/2011
Tree View to DEC
• Display Context-Based Traffic in Data Exchange Chart —
– Double-click on traffic for a tier pair in the Timeline pane to display the
corresponding context- based traffic in the Data Exchange Chart.
test
46
Graphs
• Time varying statistics
– Advanced diagnosis of network
and TCP problems
test
47
Inc.
23
2/10/2011
Graphs (cont.)
• Select graphs to display
• Options:
– Specify bucket width for
precision of graph
– Embed graphs in DEC for
easy comparison against
messages in DEC
test
– Overlay similar
graphs or
overlay all
• Select “Show”
48
Graphs: Bucket Width
• What is a bucket width?
– Statistics can be averaged over
time rather than shown as
individual data points
– A bucket is the duration of data
you want to average, like 100 ms
or 3 seconds
• Same trace, same statistics,
d e e t bucket
different
buc et widths:
w dt s:
– 100 ms
test
– 3000 ms (3 seconds)
49
Inc.
24
2/10/2011
Graphs: Embedded Statistics
• Time-aligned statistics with the Data Exchange Chart
test
50
LAB 1
LAB 1
Visualizing
Application
test
51
Inc.
25
2/10/2011
Lab 1: Visualizing Application
• Overview
– The ACE trace file represents
p
a three-tier Oracle application.
pp
You will
explore some of the different visualizations in AppTransaction Xpert that
you can use to troubleshoot and analyze the application.
• Starts on page 1 in your lab manual
test
52
Lab 1: Conclusion
AppTransaction Xpert provides many different views of a transaction
to understand the interaction of the application on the network
• Data Exchange Chart
– Shows timeline of data sent
• Tier Pair Circle
– Shows who talks to whom and summary statistics
• Tree View
– View the network packets in order, with a decode summary
– Transaction analysis
• See the timeline of transactions
• Decodes
– Drill downtest
into TCP, IP, or even application layer headers to understand
application behavior
• Graphs
– View time-based graphs of throughput, congestion, and other statistics
53
Inc.
26
2/10/2011
AppDoctor
test
What is AppDoctor … ?
• Invoke AppDoctor
– AppDoctor (menu)
– AppDoctor (icon)
• AppDoctor Tabs
– Summary of Delays
– Executive Summary
– Diagnosis
– Statistics
– Protocol-specific
test
55
Inc.
27
2/10/2011
AppDoctor: Summary of Delays
• Pie chart that breaks down the
total response time into
processing, network, and
parallel
ll l effects
ff t
• Most useful view in
AppTransaction Xpert when
troubleshooting applications
• Quickly points you to what is
the problem, or at least what is
test
definitely not the problem
• Must be used on a single
application task
56
Summary of Delays: Processing Effects
• Tier Processing delay
– Time spent processing on each tier
test
57
Inc.
28
2/10/2011
Summary of Delays: Network Effects
• Latency
– Delay caused by the physical
distance between tiers
– Common p
problem in chattyy
applications
• Bandwidth
– Delay caused by transmission
(i.e. putting the bits onto the
wire)
• Protocol
– Could be non-optimal TCP
settings
– Check Diagnosis tab for
bottleneckstest
– Check Statistics tab for values
• Congestion
– Delay caused by other traffic in
the network
58
Summary of Delays: Parallel Effects
• Delay incurred when multiple types of delay (such as tier processing,
bandwidth, and latency) occur simultaneously
test
59
Inc.
29
2/10/2011
AppDoctor – After Network Definition (16.0 )
test
BEFORE
60
AFTER
Summary of Delays: Drill-Downs
• Double-click any section of the pie chart for extra information
• Drill-downs give more information that can help diagnose problems
• Ex: Bandwidth drill-down shows throughput
g p graphs
g p and link bandwidth
test
61
Inc.
30
2/10/2011
AppDoctor: Executive Summary
• Gives overview information for application task
– Total transaction time
– Total application data transferred
– Single
Si l greatest
t t cause off delay
d l
– Tiers in application
– Bandwidth and latency at each tier
test
62
AppDoctor: Diagnosis
• Displays common
application, network,
and server problems
• Flags bottlenecks and
potential bottlenecks
• Various characteristics
of the ACE file are
compared with
thresholds
• Click on any cell
(especially on test
bottlenecks) for extra
information
63
Inc.
31
2/10/2011
AppDoctor: Statistics
• Tabular view of key
application characteristics
of” statistics quantify
• “Effect
Effect of
the components of delay
• Other statistics include
– Application message statistics
– Network packet statistics
– TCP statistics
Tip: Use it with Diagnosis
tab to
test
quantify bottlenecks
Tip: You can export to spreadsheet
64
AppDoctor: Protocol-Specific Tabs
• Protocol-specific information,
statistics, and diagnoses
• Appear automatically when
certain protocols are detected in
an ACE file:
– HTTP
– Citrix
– CORBA
– SOAP (Web Services)
test
65
Inc.
32
2/10/2011
Right-Click Operations in AppTransaction Xpert
• Graph Statistics such as throughput or congestion
• Exclude or permanently delete packets and messages
• Include ppackets and messages
g
• Rename a tier or lookup a tier name with DNS
• Re-code traffic
• Zoom in and out in the DEC
• Show protocol decodes for selected traffic
• Export decodes for selected frames in the decode viewer
test
66
Take-Away Points
AppTransaction Xpert provides an analysis of the main application
bottlenecks that is easy to use and to understand. The diagnosis is displayed
and broken down into components of delay. The different components are:
• Processing Effects
• Network Effects
–
–
–
–
Latency
Bandwidth
Protocol
C
Congestion
ti
(with recommended options)
test
• Other Effects
– Parallel Effects: Combination of any processing effects and network
effects that occur at the same time
67
Inc.
33
2/10/2011
Reports
test
Reporting Motivation
• Quick and easy to use
• Support your findings
• Recording and dissemination of all
analysis results to all interested
stake holders
• Documentation of application
behavior for future reference
test
69
Inc.
34
2/10/2011
Report Selection - Type and Format
• Very simple and quick – select report type for output (from pull down menu, or icon
button)
test
70
Report Options
• Select report options and generate report
– Report sections, title (statistics)
– Report directory (output), Report Server (optional)
test
71
Inc.
35
2/10/2011
ACE – MS Word Report
test
72
ACE – Web Report
URL
Home Dir
(all reports)
test
73
Inc.
36
2/10/2011
ACE – Multiple Transaction Report
Trans 1
Trans 2
test
74
ACE – Difference Report
test
75
Inc.
37
2/10/2011
ACE – Spreadsheet Export Reports
Comparison Report
Tree View Statistics
AppDoctor Statistics
test
76
ACE – Report Template
• Report template — Defines the previously-defined templates used
to generate the report. You can modify and customize report templates.
The standard templates:
– ApplicationDeveloperReport.ace.rt.txt
– ManagementReport.ace.rt.txt
– NetworkEngineerReport.ace.rt.txt
– concise_report.ace.rt.txt
– detailed_report_ace.rt.txt
– standard_report_ace.rt.txt
• Reports can be cataloged and managed by the Report Server
test
– Provides security,
allows access by only registered users
– Provides automatic ageing of reports
77
Inc.
38
2/10/2011
LAB 2
LAB 2
Application
Performance
Troubleshooting
test
78
Lab 2: Application Performance Troubleshooting
• Overview
– Response time problems for remote users of new web application
– Remote site connected via a 256-Kbps Frame Relay circuit
– Need to determine cause of the delay (network or web server)
– Traces have been captured at both server and client
• Objectives
– Visualize application in ACE
– Identify the cause of the application delay
– Use Transaction Analysis to identify problem with application
• Starts on page
9 in your lab manual
test
79
Inc.
39
2/10/2011
Lab 2: Conclusion
• Conclusions
–
–
–
–
–
–
–
80
The web page takes approximately 24 seconds to download
Bandwidth and congestion are the biggest sources of delay
Server delay is not a problem
Majority of traffic flow is from the server to the client
Web page consumes almost all of the 256-Kbps circuit bandwidth
Largest transaction is a download of a 352 KB graphic
This single
graphic takes 18 seconds to download
test
Performance Predictive Studies
The use cases for predictive studies:
• Predict application
pp
behavior under different network conditions
– Problematic applications
• Evaluate performance impact of application design decisions
– Modified applications
– Problematic application
• Validate infrastructure readiness for new deployments
test
81
Inc.
40
2/10/2011
Prediction in AppTransaction Xpert
• What can predictive analysis tell you?
– Response time of application under different circumstances
– Will addingg bandwidth help?
p
– Is this application sensitive to latency?
– Can we roll this out to remote users?
• What sort of studies will it help with?
– Troubleshooting performance problems
– Testing a new application
– Deploying a current application to new users
test
– Server relocation studies
82
Application Deployment Workflow
Identify performance issues
Capture and import into ACE
Create report and prediction
scenario templates
Use QuickPredict to predict
performance
test
O ti
Optional:
l U
Use M
Multi-User
lti U
QuickPredict for basic capacity
planning
Optional: Use IT Guru for
network and server capacity
planning
83
Inc.
41
2/10/2011
Application Deployment Workflow
• Optional: Use simulation for further studies
– Discrete Event Simulation to predict response times or perform server
capacity planning
– Flow Analysis for network capacity planning
– Simulation covered later in course
test
84
QuickPredict
test
Inc.
42
2/10/2011
QuickPredict
• Simplified simulation engine
• Helps study effects of the network on the application
•Q
Questions Q
QuickPredict can answer
– How much bandwidth would it take to get a 4 second response time?
– Would increasing the TCP Window improve response time?
– What happens to response time if latency increases?
– How sensitive is this application to packet loss?
– What would the response time be if the link was 65% utilized by other
traffic?
– What
h if I had
h d a 10% WAN
A Acceleration
A l i in
i my Networkk – NEW !!
test
86
QuickPredict: Outputs
• Two different output styles
– QuickPredict Bar Charts shows components of delay for different scenarios
– QuickPredict plots response time over range of network conditions
test
87
Inc.
43
2/10/2011
QuickPredict Bar Charts: Scenarios
• Click to add multiple scenarios
• Right-click to remove scenarios
test
88
QuickPredict Bar Charts: Options
• Put parameters on slider
– Drag slider back and forth to
dynamically update bar chart
• Rename scenarios
• Symmetric Bandwidth
– Set asymmetric values for
bandwidth
• Label delay categories
• Load and save parameter settings
using templates
test
89
Inc.
44
2/10/2011
QuickPredict Bar Charts: Options – NEW
Introduce WAN Acceleration
•
•
•
•
Byte Reduction
Turn Reduction
C
Compression
i
Caching
test
90
QuickPredict Bar Charts: Tabular Results
• View detailed results for each scenario in the Tabular Results tab
test
91
Inc.
45
2/10/2011
QuickPredict: Response Time Graphs
• Shows sensitivity of task response time to network characteristics:
– Bandwidth
– Latencyy
– Packet Loss
– Link Utilization
– TCP Window Size
– WAN Acceleration
test
92
QuickPredict: Advanced View
• Choose network path to modify
• Choose an X-axis parameter and the range of values over which to plot
response times
• Select values for each of the other parameters
• Click Update Graph
• Overlay multiple curves on same graph
• Load and save templates
test
93
Inc.
46
2/10/2011
QuickPredict: Table View
• Modify parameters on all network paths in the task
– Red text indicates X-axis parameter
– Use Set X-axis to specify
p
y X-axis p
parameter
– Load and save templates
test
94
QuickRecode
test
Inc.
47
2/10/2011
QuickRecode
• Changes high level application characteristics
– Shows sensitivity of task response time to application characteristics:
• Application turns
• Total application bytes sent
• Total processing time in tiers
– Use with QuickPredict to see how response times are affected
test
96
QuickRecode: Use Cases
• Use Cases
– Performance benefits
– Changing database access
to reduce chattiness
– Demonstrate effect of faster
app server
Hint: Use QuickRecode to
prove effect of optimizing
application
test
97
Inc.
48
2/10/2011
LAB 3
LAB 3
Predictive
Analysis
test
98
Lab 3: Predictive Analysis
• Overview
– Deploying a new application to several remote locations
– Application has client querying an Oracle database
– Application has been deployed to pilot site (in same city as database) and
poor performance is already apparent
– Goal is to determine if the application deployment will be successful
– If not, what can be done to improve performance?
• Objectives
– Perform a deployment study using QuickPredict Bar Charts
– Use AppTransaction Xpert to drill down into the app and see what’s going
on
test
– Use QuickRecode
to test a possible fix
99
Inc.
49
2/10/2011
Lab 3: Conclusion
• Always try to determine “goal” or “objective” of any problematic application
– Is there an SLA expectation
– What is the user expectations
– What are other users, departments, experiencing
• Additional useful troubleshooting information to determine
– Is the problem reproducible or is it transient
– Are all transactions of the application behaving the same
test
100
Agenda
 Capture Methodologies
 Capture Data Sources
- New Capture Manager Options
 Usingg AppTransaction
pp
Xpert
p
to p
perform p
packet captures
p
- Capture Types
- Lab 4: Capture Process
 Using AppTransaction Xpert to filter packet captures
- Capture Filters
 Using AppTransaction Xpert to import packet captures
- Import Process
- Lab
L b 55: Identifying
Id tif i P
Performance
f
P
Problems
bl
in
i Unfiltered
U filt d Traces
T
test
101
Inc.
50
2/10/2011
Capture Methodologies
test
Basic Capture Methodology
• Define and understand the problem at hand
– What is slow?
– Is it the login or another specific task?
– How often does it occur?
• What is a task?
– Isolate an “atomic” Transaction
– Loading single web page
– From when user clicks “Submit” to screen refresh
• Plan your capture
– Simultaneous or single-sided
– Which tiers are involved in the application?
• Aim for as “clean”
test a packet trace as possible
–
–
–
–
103
Keep measurement as simple as possible
Filter out unrelated traffic
Capture one occurrence of one task at a time
Perform capture numerous times to help eliminate anomalies
Inc.
51
2/10/2011
Atomic Transaction Example (Task)
• Web Application of a Bank
– Load Login Page
• Open a Web Browser
• Type
yp URL of the Bank
• “Enter”
• Login Page Loads
– Login
• Type User credentials
• “Login”
• Home Page shows up
– Check Savings Account
• Mouse/cursor around
• Click on “Savings
Savings Account
Account”
• Savings Accounts
Page Loads
test
– Logout
• Call up Logout screen
• “Submit”
• Logged out page
104
Red arrow = Start Capture
Blue arrow = Stop Capture
Capture Data Sources
test
Inc.
52
2/10/2011
Capture Data Sources
• Packet traces are the primary data inputs
• AppTransaction
pp
Xpert
p
supports
pp
a wide variety
y of packet
p
sources
–
–
–
–
ACE Capture Agents
ACE Live
ACE Enterprise Management Server
Agentless Capture Options – NEW!
• F5 Load Balancers
• UNIX, Linux platforms
• WAN Accelerators
test
• Cisco NAM
– Wireshark and other Winpcap and Libpcap sources
• TCPDump, Snoop
106
Agentless Capture Options – NEW
• No software installation required
• Use the AppTransaction Xpert Capture Manger to control captures remotely
via SSH
• Supported on platforms that support TCPDump
– TCPDump file remains on machine until capture is completed, then deleted after
retrieval
• Included in the AppTransaction Xpert Plus license
test
107
Inc.
53
2/10/2011
Agentless Capture Options – NEW (cont.)
• F5 Load Balancers
– Excellent way to follow a transaction to the proper server
– Perform successful trace file merging across firewalls
– Rule out the device as a source of delay
• UNIX-Based Operating Systems
– Supported Platforms
• Linux
• Solaris SPARC
• Solaris Intel/AMD
• AIX
• FreeBSD
test
• HP-UX Itanium
• Macintosh PowerPC
• Macintosh Intel
• True64
108
Agentless Capture Options – NEW (cont.)
• WAN Acceleration Devices
– Support for Riverbed and Cisco
– Restore visibility of the WAN
– More information can be found in session 1471 - Monitoring and
Troubleshooting Applications in a WAN Accelerated Environment with
ACE Live™ and AppTransaction Xpert
test
109
Inc.
54
2/10/2011
Capture Data Sources – AppResponse Xpert
(ACE Live)
• New seamless 1-click integration
– Let ACE Live do the filtering for you
• Easily pull traces from alert snapshots
off actual
t l application
li ti issues
i
test
110
Capture Data Sources – AppForensics Xpert
(ACE EMS)
• ACE Enterprise Management Server
– Provides web-based extension to Continuous Capture functionality
– Supports both End-User Driven Capture workflows, as well as Server-Driven
Capture Workflows
test
– End Users report application slowness
• Trouble ticket is created with trace files of the incident attached
• Troubleshooters claim problem reports and troubleshoot on their schedule
111
Inc.
55
2/10/2011
AppForensics Xpert – Web Capture Retrieval
test
112
Capture Data Sources – AppForensics Xpert
• Server-Driven Capture Workflow
– Panorama provides forensic actions based on threshold violations within the
application servers
– The trace file forensic action automatically generates a trace that is accessed
through AppForensics Xpert
– For more detail, attend session 1482 – Deploying an Enterprise-Wide Packet
Capture Architecture with ACE EMS and ACE Live
test
113
Inc.
56
2/10/2011
Capture Data Sources – ACE Agents
• Benefits of using ACE Agents
– Unlicensed
– Support for major platforms (Windows, HP-UX, AIX, Solaris, Linux, Mac OS X,
FreeBSD)
– Lightweight, Secure, and Stable
– Silent Install capability
– Server Statistics collection (Windows)
– Enhanced Filtering (NEW)
– On-Demand and Continuous Capture modes
test
114
Additional Data Sources
• AppTransaction Xpert can import a variety of additional data
– Identify these sources during capture to ensure collection
• From OPNET products
– Windows ACE Capture Agents can collect Lightweight Server Statistics
– OPNET Panorama
• Server statistics and performance metrics
• Java transaction tracing
• Third-party sources
– HP GlancePlus
– Perfmon
– Custom datatest
in comma
saved (.csv) format
115
Inc.
57
2/10/2011
Capture Lightweight Server Statistics
• Capture Windows performance data
– ACE Capture Agent generates two files
– Application traffic (*.appcapture) file
– Windows performance data (*.appcapture.da) file
 Display internal server metrics
• Percent Processor Time
• File Read Bytes Per Second
• System Calls Per Second
• Available MB
• File Write BytestestPer Second
116
AppInternals Xpert (Panorama) Server Statistics
• Capture Server Statistics from AppInternals Xpert
– Use with on-demand captures
– Thousands of available metrics
• For more information:
– Session 1461- OPNET Panorama® Essentials: Multi-Tier Application Monitoring,
Correlation, and Troubleshooting
test
117
Inc.
58
2/10/2011
AppInternals Xpert Transaction Trace
• AppInternals Xpert can create a trace of all activity through the Java
Virtual Machine (JVM) or .NET CLR
– Provides a call-by-call sequencing (or call stack) of all activity within your
application
li i
• This transaction trace can be imported into AppTransaction Xpert to explain
what is occurring on a given system during long processing delays
test
118
Capture Manager
• Capture Manager control parameters
test
119
Inc.
59
2/10/2011
Capture Agent Settings
test
120
Capture Types
test
Inc.
60
2/10/2011
On-Demand Capture
• Methodology
–User knows exactly when a problem is going to occur and
what it takes to reproduce it
–Deploy and configure ACE Capture Agents
–Start the capture
–User re-creates the problem
–Stop the capture
Import capture for analysis
–Import
test
122
On-Demand Capture (cont.)
test
123
Inc.
61
2/10/2011
On Demand Capture Process: Adding an Agent
test
124
On Demand Capture Process: Specify Agent
Settings
test
125
Inc.
62
2/10/2011
Capture Process: Start Capture
test
126
Capture Process: Finish Capture
test
127
Inc.
63
2/10/2011
Capture Process: Open in Transaction Analyzer
test
128
Scheduled Capture
• Useful for reproducible problems during off hours
• Methodology
–User knows when a problem will occur
• For example, after-hours batch process or database backup
–Schedule a capture
• Choose one-time or repeating capture
• Set the schedule for a capture
• Specify schedule for download
–Import
I
t trace
t
for
f analysis
l i
test
129
Inc.
64
2/10/2011
Scheduled Capture (cont.)
test
130
Scheduled Capture (cont.)
Scheduled capture can create one-time,
daily, weekly, or monthly capture
schedules
test
Scheduled capture utility can optionally
create a one-time, daily, weekly, or
monthly schedule to download trace files
131
Inc.
65
2/10/2011
Continuous Capture
• Useful for troubleshooting intermittent problems
• Methodology
–User experiences intermittent problem
–Start continuous capture
–User calls help desk
• Problem occurred again around 9:30
–Preview capture
–Download
Download trace of problematic time span
test
–Import trace
for analysis
• Agent details
–Same capture agents as on-demand capture
132
Continuous Capture (cont.)
test
133
Inc.
66
2/10/2011
Continuous Capture: Preview/Download
test
134
LAB 4
LAB 4
Capture
Process
test
135
Inc.
67
2/10/2011
Lab 4: Capture Process
• Overview
– Configure a capture agent
– Perform an on-demand capture
p
– Open in Transaction Analyzer
• Skills covered
– Learn how to configure a capture agent
– Understand how to use agent filters
– Capturing from the network
test
136
Lab 4: Conclusion
• Configured capture agent for capture
– Located and specified host IP Address
– Specified capture mode
– Set up a filter
• Performed a capture
– Started the agent
– Performed a transaction
– Stopped the agent
– Brought
g the capture
p
file into Transaction Analyzer
y
test
137
Inc.
68
2/10/2011
Capture Filters
test
Filtering Unrelated Traffic
• You want to capture the transaction of interest
– AppTransaction Xpert lets you filter out irrelevant traffic
• Options to filter during capture, during the import or after the import
– The earlier the better
• Why filter during capture?
– Only capture the specific transaction
– Smaller trace file to process => easier/faster import
• Why not filter during capture?
– Sometimes you want to see the whole picture
– If you’re not 100% sure about application path
• Recommended
test methodology
– Capture without filters for short duration
– Examine trace
– Determine best filter to use
– Capture subsequent tasks with filter
139
Inc.
69
2/10/2011
Filter: During Capture
Users can filter out or only allow traffic based on a specific protocol
Users can filter specific hosts
and/or ports for incoming or
outgoing traffic
 New Filtering Options in
Capture Agent test
3.9




140
Use wildcards and slash notation
Use DNS names
Port ranges
Broadcast and multicast can be
excluded independently
Users can elect to exclude or
allow broadcast and multicast
traffic
Filter: During Capture (cont.)
• Preview in Continuous Capture
– Select the IP addresses, conversations and time ranges of interest
• Great for troubleshooting intermittent issues
test
141
Inc.
70
2/10/2011
Filter: During Import
• What is Trace Explorer?
– Graphical protocol analyzer
– Used to helpp filter unrelated traffic
during import process
• Why would you use it?
– Graphically filter a capture file
– Allows for fast read of large trace
files
– You want to know what you’re
looking at
test
142
Filter: After Import
• Once capture file has been imported, users
can still filter irrelevant traffic
– Right-click on any tier pair
conversation, application message,
application
li i transaction
i or networkk
packet
• Option to Exclude or Permanently
Delete
– Filter Wizard
• Remove unidirectional, non-IP, UDP,
broadcast traffic
– Capture File Filter
• Use to remove traffic by host, port,
protocol
143
test
Inc.
71
2/10/2011
Import Process
test
Import Overview
• Now that the data has been captured, it must be imported into
• Review what has been captured
– Do you have a single capture file?
– Do you have more than one capture file?
– Are you capturing WAN accelerated traffic?
– Is the capture file filtered?
• If not, use Trace Explorer
• Greatly simplified
for 16.0 !!
test
145
Inc.
72
2/10/2011
Import Changes in 16.0
• Import process has been greatly simplified
– 15.0 required answering a series of questions about tier locations, bandwidths, and
latencies
– For 16.0, we focused on asking as few questions as possible
• Filter, view statistics or diagnoses, or explore your transaction immediately
• Locations, bandwidths, and latencies can still be specified to break down Network
Transfer effects into Bandwidth, Protocol, and Congestion
• Most importantly, you can change these values whenever you need to
• Import is now broken down into use cases
– Single packet trace
– Merge several traces
– Merge traces
from a WAN-accelerated environment
test
– Batch import
– Filter into Trace Explorer
146
Import a Single Capture File
test
147
Inc.
73
2/10/2011
Merging Capture Files
• What is a merge?
– Two or more traces of an application transaction can be combined
– Creates unified view of application
– Accurately determines the send and receive times of each packet
• Benefits and use cases
– Allows analysis of multi-tier transactions where all traffic does not
go through one location
– Improves analysis in a production environment with congestion issues such as
packet loss and retransmissions
test
148
Merging Capture Files (cont.)
• All traces must be of the same transaction
• All must have been captured at the “same time”
• Easier to import
p if there is some common traffic between traces
test
149
Inc.
74
2/10/2011
File name
test
Load previously saved import
details (tier names, tier locations,
bandwidth, latency,…)
150
• Replace IP addresses with friendly names (can use DNS)
test
151
Inc.
75
2/10/2011
After Opening/Merging
• Now what?
– Filter down to a single transaction or remove user think time
– Open AppDoctor
• Note that there are only Processing and Network effects identified
– Specify bandwidth and latency to break out individual network effects
test
152
AppDoctor
Bandwidth, protocol and
congestion are all
included in one category
named “Network
Transfer”
test
153
To break down the various
components
t off “Network
“N t
k
Transfer”, click the
“Refine Network
Effects…” button
Inc.
76
2/10/2011
Refine Network Effects
Specify the local and
remote bandwidths
here
test
154
Tier locations and
latencies are
automatically
detected, but can be
changed if needed
Specify Bandwidth and Latency
• Minimum bandwidth
– Capacity of the slowest link
– Should be a known factor about your network
• Minimum one-way latency
–
–
–
–
PathProbe provides an estimate
Half the value returned by ping
Auto-calculated in a trace merge
In manyy cases,, AppTransaction
pp
Xpert
p
can estimate latencyy of a single-sided
g
trace
test
155
Inc.
77
2/10/2011
Specify Bandwidth and Latency (cont.)
• Quiz
– Bandwidth of each segment is known
– Ping reports 46ms RTT
test
• What would you specify for bandwidth and latency between the Client and
the Server during import into AppTransaction Xpert ?
Bandwidth = 1544 Kbps
156
Latency = 23ms
AppDoctor – After Network Definition
test
BEFORE
157
AFTER
Inc.
78
2/10/2011
LAB 5
LAB 5
Identifying
Performance
Problems in
Unfiltered Traces
test
158
Lab 5: Identifying Performance Problems in
Unfiltered Traces
• Overview
– Open a single side capture
– Filter traffic
– Refine network effects
– Analyze transaction
• Skills covered
– Step through the import process in AppTransaction Xpert
– Understanding the importance of bandwidth settings
test
159
Inc.
79
2/10/2011
Lab 5: Conclusion
• Imported a single-sided capture file
– Imported capture file
– Filtered irrelevant traffic
– Estimated locations
– Entered local and remote bandwidth
– Estimated latency
test
160
Introduction to AppTransaction
Xpert
test
Day 2
Inc.
80
2/10/2011
What to Expect in Advanced AppTransaction
Xpert Topics
• Mix of lecture and lab
• Advanced class
– Address network concepts important to using AppTransaction Xpert
– Dive into “the science” behind ACE
– Highlight new ACE features
– Discuss TCP
• Please be interactive
– Participate test
– Ask questions
162
Prerequisites
• Prerequisites
– Reviewed yesterdays material
– Reviewed session 1441
– Equivalent experience
• You should already understand the mechanics of ACE
– Capturing a trace
– Importing traces
– Generating reports
• You should already
recognize “vanilla” problems
test
– Chatty applications
– Insufficient bandwidth
– Slow servers
–…
163
Inc.
81
2/10/2011
Agenda
 How AppTransaction Xpert Calculates and Predicts Components of Delay
- Lab 1: Calculating Network Delays
 Understanding the different Components of Network Delay
- Congestion Delay
- Protocol Delay
- Lab 2: Understanding Components of Delay
 Understanding Parallel Effects
- Parallel Effects
- User Think Time
test
- Lab 3: Understanding Parallel Effects
164
How AppTransaction Xpert
calculates and predicts
p
of Delayy
components
test
Inc.
82
2/10/2011
Components of Application Delay
• The following are key components of delay in a transaction:
– Processing
• Client
• Server(s)
– Network
• Protocol (TCP WIN, Nagle’s, etc.)
• Bandwidth
• Latency (baseline, dominated by network hardware and ultimately ruled by C)
• Congestion (including queuing)
• Packet loss
– Application itself
test
• Think of the top two topics as “resources”. How does the app use them?
– Efficient/inefficient code
– App turns/Chattiness
166
Review of Network Delays
• Packets experience a complex series of
– Network hops
– Technologies
– Queuing delays
–…
T1
OC3
test
167
Inc.
83
2/10/2011
Apps See the Network as a “Black Box”, So Does
• For most troubleshooting, complexity can be simplified
• Think of entire network as a single “equivalent” circuit
• Once packet leaves source, the delay can be separated into 3 categories
– Bandwidth delay
– Latency delay
– Congestion (queuing) delay
• What about Protocol?
– Protocol delay is a function of the sender/receiver
T1
OC3
– Packet Loss is handled by the protocol
test
Bandwidth, Latency,
Congestion
168
Bandwidth Delay
• Time it takes to stream all bits onto circuit
• This number varies with packet size
• Can be calculated from ppacket size and circuit bandwidth
2000 bits
Headers
Data
2000 bits/second
…01110010010
01110010010
test
 What is the bandwidth delay for this packet?
169
Inc.
84
2/10/2011
Calculate Serialization Delay
How long does it take
to serialize this packet
on to the wire?
Serialization and BW
are the same thing
• If you know the bottleneck bandwidth across a link and the size of each
test
packet, you know how long it takes to get across the link due to serialization
delay (ie: Bandwidth delay)
• Note: we are ignoring latency at this point
170
Latency Delay
• Time it takes for signal (1st bit) to travel across network
– Distance / speed of light
– Normal delays caused by device forwarding
• This is a constant number for all packets
7 ms
1st bit
10 ms
1 ms
10 ms
1 ms
10 ms
1 ms
 What is the latency
for this circuit?
test
 What value would “ping” produce?
171
Inc.
85
2/10/2011
Application Turns
• A turn is a change in the flow direction of application data
• Turns magnify circuit latency
• An application
pp
with no turns would onlyy experience
p
latency
y once
• An application with 99 turns would experience latency 100 times
• Simple equation for total latency delay
– Latency Delay = circuit latency * (turns +1)
– This equation only works on sequential applications
No Application
pp
Turns
Manyy Application
pp
Turns
test
172
Congestion Delay
• Queuing on devices
• Usually caused by other traffic
• Can varyy from packet
p
to packet
p
Forwarding rate = 1 packet / ms
packet
test
5 packets in forwarding queue
 What is the congestion delay for this device?
 What is the total delay across this device?
173
Inc.
86
2/10/2011
Determine Queuing/Congestion
Packet should have taken
thi time
this
ti . . .
. . . but ended up
taking this time
test
• If a packet should take x time but takes y time, y – x yields
queuing/congestion delay
174
Example Network Delay
• Bandwidth = 1,000 bps
• Latency = 5 seconds
T=0
10
20
30
40
50
Client
Bandwidth delay
30 seconds
30,000 bits
Latency delay
5 seconds
Congestion delay
test
Server
15 seconds
50
175
Inc.
87
2/10/2011
Packet Trains
• Stream of packets that are sent together
• Arrive at destination spread out
• Manyy TCP segments
g
of a larger
g application
pp
message
g
• Spreading caused by limited bandwidth of bottleneck link
T=0
10
20
30
40
50
60
10
20
30
40
50
60
test
0
176
Calculating Delay for Packet Trains
• Treat packet trains like 1 big packet
• Same rules apply
– First—establish delay caused by bandwidth (stream bits onto wire)
– Second—add latency (last bit of the packet train getting across)
– Third—anything left over is congestion
Application Layer
Transport Layer
test
Network Layer
177
Inc.
88
2/10/2011
Calculating Delays—Advanced
• Bandwidth = 1,000 bps
• Latency = 5 seconds
T=0
10
20
30
40
50
60
70
80
90
100
Client
Bandwidth Delay
5 + 10 + 30
Latency Delay
5 +5 +5
Congestion Delay
5 + 15
Tier Processing (Client)
10
test
10
5,000 bits
20
Server
65
80
Tier Processing (Server)
10
100
10,000 bits
Total Delay = 45+15+20+10+10 = 100
178
Pie Chart
• AppTransaction Xpert creates a pie chart that shows all of the components of delay in one
screen
• Shows the total impact on response time due to each individual effect
• Each wedge is the maximum benefit that a single upgrade will provide
Bandwidth
45
Bandwidth
Latency
15
Latency
Congestion
20
Client
10
Server
10
Congestion
Client
Server
test
179
Inc.
89
2/10/2011
Network Effects in App Doctor
• AppTransaction Xpert 16 does not require fully specifying network data for
import
• Creates a “Network Effects” pie slice
test
180
Network Effects
• Specifying Bandwidth will cause AppTransaction Xpert to recalculate the
values for the components of delay
test
• For more information, see
Session 1442: Application Capture
and Import Strategies with AppTransaction Xpert
181
Inc.
90
2/10/2011
Calculating Delays
• Two ways to calculate the Pie Chart
– Adding up (add the components of delay)
– Squeezing down (reduce effects one at a time)
T=0
10
20
30
40
50
60
70
80
90
100
Client
test
Server
10
182
20
65
80
100
Review of Network Delays
• Different factors can affect network delay
– Bandwidth
– Latency
– Congestion
• Some factors are constant and some are variable
• Network delay is not always due to lack of bandwidth
• Important that you understand fundamentals to interpret ACE results
test
183
Inc.
91
2/10/2011
LAB 1
LAB 1
Calculating
Network
Delays
test
184
Lab 1: Calculating Network Delays
• Overview
– Apply knowledge of sources of delay to a real application
• Objectives
– From the raw statistics calculate total bandwidth delay
– From the raw statistics calculate total latency delay
– Compare results with ACE analysis
test
185
Inc.
92
2/10/2011
Lab 1: Conclusion
 Response Time = 26.03
 Network Data = 3,199,760 Bytes
 Size of link = 1,544,000 Kbps
p
 Bandwidth Delay = (3,199,760*8)/1,544,000 = 16.58
 Percentage of Bandwidth = 16.58/26.03 = 63.7%
 Turns = 337
 Latency = 24.23ms
 Latency Delay = 24.23*(337+1) = 8.18 seconds
 Percentage
P
t
off L
Latency
t
= 8.18/26.03=
8 18/26 03 31.4%
31 4%
test
186
Lab 1: Conclusion
 Calculated that Bandwidth caused 63.7% of response time
 Calculated that Latency caused 29.8% of response time
test
187
Inc.
93
2/10/2011
Lab 1: Conclusion
 Why didn’t the math work out perfectly?
 Some amount of congestion in trace
 Some amount of protocol delay in trace
 Equation
E ti didn’t
did ’t accountt for
f bi-directional
bi di ti l traffic
t ffi
 Our simple math was very close
 Important to perform this type of “sanity check”
test
188
Congestion Delay
test
Inc.
94
2/10/2011
Congestion Delay
 It is somewhat amazing that congestion can be calculated
 Very difficult to measure and calculate manually
 ACE can measure the effect even without seeing the cause
 ACE provides 2 important pieces of information for
congestion
 Actual congestion delay for each packet
 The cumulative effect of congestion on the response time
test
190
Key Concept
 Trace file includes 1 piece of time information
 Time the packet arrived at the capture probe
1 piece of time information per packet
 ACE includes 2 pieces of time information for each packet
 Time the packet left the sending side
 Time the packet arrived at the receiving side
test
2 pieces of time information per packet
191
Inc.
95
2/10/2011
Key Concept
 ACE knows both the send and receive time
 Delay analysis requires that both values are known
 How does ACE derive the second ppiece of time information?
 Trace Merge
 Single Trace Adjust
test
192
Trace Merge
 Each trace shows what happened locally
 Once clocks are aligned, draw the packets from send to receive
Trace captured at client
side
T = 0 1 2 3 4 5 6 7 8 9 10
Client
T = 0 1 2 3 4 5 6 7 8 9 10
Client
A
BC
C*
A
B C
C*
D
D
Trace captured at server
test
side
AB
C* D
A B
Server
C* D
Server
0 1 2 3 4 5 6 7 8 9 10
0 1 2 3 4 5 6 7 8 9 10
193
Inc.
96
2/10/2011
Single Side Adjust
 With a single trace, ACE only knows what happened locally
 Must estimate what happened on network and at other side
 Based on bandwidth and latency values set during import
 ACE estimates
ti t when
h th
the packet
k t was sentt
 ACE estimates when the packet will arrive
T = 0 1 2 3 4 5 6 7 8 9 10
Client
A B
C
C*
E
Trace captured at client
side
T = 0 1 2 3 4 5 6 7 8 9 10
Client
A
BC
test
C*
E
Server
0 1 2 3 4 5 6 7 8 9 10
194
Trace Merge
 What happens if the import parameters are incorrect
 Bandwidth is set too low
g
 Latencyy is set too high
 Might create false packet crossings
 This will confuse analysis
T = 0 1 2 3 4 5 6 7 8 9 10
Client
A B
C
D
E
Trace captured at client side
T = 0 1 2 3 4 5 6 7 8 9 10
Client
A
BC
D
test
E
Server
0 1 2 3 4 5 6 7 8 9 10
195
Inc.
97
2/10/2011
Key Concept
 With BOTH send and receive time ACE knows
 Actual packet delay
packets were sent and received
 Order of how p
 What would happen if we graphed the delays?
testTotal
196
Packet Delay
Packet Size vs. Delay Graph
Delay (sec)
• Graph the packet size vs. total packet delay
Congestion
Slope is bandwidth*
test
Pure latency
Packet Size (bits)
* Inverse of bandwidth (1/BW)
197
Inc.
98
2/10/2011
Packet Size vs. Delay Graph (cont.)
Delay (sec)
• Graph Packet Size vs. Total Packet Delay
test
Packet Size (bits)
198
Viewing Congestion in ACE
• Several good ways to view congestion in ACE
– AppDoctor diagnosis
– Summary of Delays
– Congestion drill-down
– Data Exchange Chart
test
199
Inc.
99
2/10/2011
Congestion in the AppDoctor Diagnosis
• Specifically marks category as a Bottleneck
test
200
Congestion in the Summary of Delays
• Shows the cumulative effect of the congestion on response time
• How much slower was the application because of congestion?
test
201
Inc.
100
2/10/2011
Drilling into Congestion
• Drilling into congestion can show if the packets are getting queued or
test
otherwise slowed down
202
Drilling into Congestion (cont.)
• Look for “plateaus”
test
• Always keep in mind the scale of the chart!
• Is delay uniform or random?
– Is there any encryption on the link?
203
Inc.
101
2/10/2011
Congestion in the Drill-Down
• Packet Congestion graph
• Retransmission graph
test
204
Congestion in the DEC
• Network chart layered with Packet Congestion graph
test
205
Inc.
102
2/10/2011
Congestion Review
• Congestion Delay is cause by other traffic on the network
• Always better to merge traces from congested circuits
– Congestion values for a trace merge are precise
– Congestion values for a single-side adjust are approximations
• Several ways to see congestion and its effect on the application
• Imagine “squeezing” the congestion out of the trace to calculate its
contribution to overall delay
test
206
Protocol Delay
test
Inc.
103
2/10/2011
Protocol Delay
• Caused by the artificial throttling of traffic
• Packets are prevented from even getting onto the network
• Caused mainlyy byy TCP
test
208
Why Does TCP Exist?
• TCP offers reliable data transport to applications
– Application data is guaranteed to be delivered
– Packets that are “lost” by the network will be retransmitted
• TCP makes applications behave in a network-friendly manner
– Throttles the rate at which application sends data
– Ensures that no single user consumes all of the bandwidth
• Both can impact application performance
test
209
Inc.
104
2/10/2011
What Does Protocol Delay Look Like?
• Protocol delay prevents packets from even getting on the network
Application Layer
Transport Layer
Network Layer
No Protocol Delay
Protocol Delay
test
Protocol
Delay
0 1 2 3 4 5 6 7
210
0 1 2 3 4 5 6 7 8 9 10 11 12 13
Observe Protocol Behavior
Protocol delayy is when the
sending side delays or
throttles sending of data (this
is different than processing
delay)
• Observe how data leaves in stages indicating Protocol Delay
test
• Observe other attributes like TCP window size
• Our example here is ridiculously simplified, but real protocol delay is
observable in a real trace
211
Inc.
105
2/10/2011
Forms of Protocol Delay
• Critical that you understand mechanisms of TCP
• Several different TCP mechanisms cause can protocol delay
– TCP windowing
– Slow Start
– Frozen window
– Nagle’s algorithm
– Retransmissions
– Out of sequence packets
test
212
Viewing Protocol Delay in ACE
• Several ways to view Protocol Effects
–AppDoctor diagnosis
–Summary
S
off D
Delays
l
–Protocol drill-down
–TCP Visualization in DEC
test
213
Inc.
106
2/10/2011
Protocol Delay in the AppDoctor Diagnosis
• Specifically marks category as a Bottleneck
test
214
Protocol Delay in the Summary of Delays
• Double-click on slice to get more detail
test
215
Inc.
107
2/10/2011
Protocol Delay Drill-Down
• Graphs dynamically appear based on AppDoctor Diagnosis
• Users can manually add or remove graphs
• Protocol Delayy
– TCP In-Flight Data
– Retransmissions
– Out of Sequence Packets
– TCP Nagle’s Delay
– TCP Advertised Receive Window
test
216
Protocol Delay Review
• Protocol delay results from the artificial throttling of traffic
• Several different TCP mechanisms can cause it
• Prevents the ppacket from even getting
g
g onto the network
• Can often be calculated without a trace merge
• Imagine “squeezing” the protocol effects out of the trace to calculate its
contribution to overall delay
test
217
Inc.
108
2/10/2011
LAB 2
LAB 2
Understanding
Components of
Delay
test
218
Lab 2: Understanding Components of Delay
• Overview
– User is transferring files with Microsoft file browser
• Transfer time is very slow
• Want to understand cause of delay
• Objectives
– Identify and understand the sources of delay
– Compare slow transfer to fast transfer
• Homework
test
– Continue with homework sections of lab manual if you have time
219
Inc.
109
2/10/2011
Lab 2: Conclusion
• File transfer is slow
• Primary source of delay is protocol/congestion
• Some “potential” problems with TCP but no smoking gun
• Look at packet delay graph to evaluate congestion
• Significant congestion on WAN circuit
– Base latency is 25ms
– Some packets take 125 ms to cross the network
test
220
Lab 2: Conclusion
• Congestion is primary cause of delay
• Removing other traffic on network fixes problem
– Reduces congestion
– Response time decreases
• Might be a case for QoS
Significant Congestion
No Congestion
test
221
Inc.
110
2/10/2011
Parallel Effects
test
The “Mysterious” Gray Wedge
• AppTransaction Xpert shows “parallel effects” as a gray pie wedge
– Transaction has more than one effect taking place at one time
test
– HTTP is an example of a transaction that exhibits parallelism
223
Inc.
111
2/10/2011
Parallel Effects Drill-Down
• Can show the individual effects with their combined parallel effects together
• Solid bars are the individual effects
• Striped
p bars are the parallel
p
effects
• Optimizing both components simultaneously will give the benefits of the
individual effects AND the parallel effects
T=0
10
20
30
40
50
60
70
80
Bandwidth
60%
Latency
10%
Server
20%
Parallel
10%
Response Time
90
100
Parallel Effects of Bandwidth and Latency
test
Parallel Effects of Server and Latency
224
Understanding Parallel Effects
• Parallel delays may seem counterintuitive at first
• It is a real effect and does happen in the real world
• Real Life Parallel Effects example
p that yyou will remember!
– Reading a paper while eating breakfast
– Cooking a meal (dessert in the oven while steak is on the grill)
• Would be difficult to split these into simple 2 category pie charts
– Incorrect to say grilling steak was 50% and dessert was 50% ?
– Suggests that eliminating dessert would cut dinner preparation in half
– Probably more like 25% steak, 25% dessert, 50% both
– Life is parallel
test
225
Inc.
112
2/10/2011
Types of Applications
• Applications fall into two main categories
–Transactional
• Strong
St
cause and
d effect
ff t relationship
l ti hi bbetween
t
packets
k t
• Each application message is dependent on the one before it
• Examples
– Database queries
– Web transactions
– File transfers
–Asynchronous
• No strong
cause and effect relation between packets
test
• Examples
– Voice
– Citrix
226
• Transactional application can be further divided
–Sequential
• Single
Si l execution
ti path
th
• Application is only doing one thing at a time
–Parallel
• Two or more simultaneous execution paths
• Application is doing multiple things at a time
test
227
Inc.
113
2/10/2011
• Sequential (Adding up method)
– Each slice of time can be placed into a single delay category
– The components of delay add up to the total response time
T=0
10
20
30
40
50
60
70
80
90
100
Client
test
Server
228
• Parallel (Squeezing down method)
– Some slices of time could be placed in multiple delay categories
– The components of delay would up to more than the total response time
T=0
10
20
30
40
50
60
70
80
90
100
Client
test
Server
Connection 1
Connection 2
Connection 3
229
Inc.
114
2/10/2011
AppDoctor Individual Effects
• Summary of Delays for a sequential transaction
Response Time
6 sec
Server
2 sec (33%)
Client
2 sec ((33%))
Bandwidth
2 sec (33%)
test
230
AppDoctor Parallel Effects
• Summary of Delays for a parallel transaction
• Server is now processing while simultaneously sending the large message
test
231
Inc.
115
2/10/2011
AppDoctor Parallel Effects (cont.)
How does AppDoctor handle multiple effects occurring at the same time?
test
The "Parallel Effects" category
Response
Time
6 sec
Server
2 sec
(33%)
Client
2 sec
(33%)
Parallel
Effects
2 sec
(33%)
Mixed wedge shows total impact due to server processing and bandwidth occurring in parallel
Because there may be many mixed wedges, ACE shows these as gray in the Pie Chart
232
Understanding Parallel Applications
• Calculating delays for sequential transactions is fairly easy
– Each slice of time can be attributed to a single cause
– All the slices add up to the total
• Calculating delays for parallel transactions can be complex
– A slice of time may be attributed to multiple sources
– Only the squeezing method works here
• Parallel Effects are subtle and not always easy to visualize
• Parallel Effects do not always occur just because delays overlap
• Aren’t you glad you have AppTransaction Xpert to do this for you 
test
233
Inc.
116
2/10/2011
Parallel Effects Example
• Parallel
– Two independent connections occur in parallel
– Both must complete for the application to complete
Client
Conn. B
B
B
B
A
Conn. A
test
Server
234
• Slide Conventions
– Show independent connections as parallel DECs
– Keep in mind that these are happening on top of each other
– Obvious now why you can’t just add delays up
T=0
10
20
30
40
50
60
70
80
90
100
Client
Connection A
Server
T=0
10
20
test
30
40
50
60
70
80
90
100
Client
Connection B
Server
235
Inc.
117
2/10/2011
• Parallel applications with squeezing method
– Perform experiments that squeeze out each category of delay
• Color associated with that delay would disappear
• Response time of transaction may decrease
– Record the difference from each experiment
T=0
10
20
30
40
50
60
70
80
90
100
Client
Server
T=0
10
20
test
30
40
50
60
70
80
90
100
Client
Server
236
• What would happen if bandwidth became infinite?
– Green would be squeezed out
– Total response time of the transaction would change to 40 seconds
– Bandwidth is responsible for 60 seconds of response time (60%)
T=0
10
20
30
40
50
60
70
80
90
100
Client
Server
T=0
10
20
test
30
40
50
60
70
80
90
100
Client
Server
237
Inc.
118
2/10/2011
• What would happen if latency became zero?
– Red would be squeezed out
– Latency is responsible for 10 seconds of response time (10%)
T=0
10
20
30
40
50
60
70
80
90
100
Client
Server
T=0
10
20
test
30
40
50
60
70
80
90
100
Client
Server
238
• What would happen if the server became infinitely fast?
– Yellow would be squeezed out
– The server is responsible for 20 seconds of response time (20%)
T=0
10
20
30
40
50
60
70
80
90
100
Client
Server
T=0
10
20
test
30
40
50
60
70
80
90
100
Client
Server
239
Inc.
119
2/10/2011
• Something interesting just happened
– Bandwidth caused 60%, Latency caused 10%, Server caused 20%
– The individual sources of delay are only responsible for 90%
– What caused the remaining 10%?
• Delays that are not removed by a single source are Parallel Effects
Bandwidth
60%
Bandw idth
Latency
test
240
Server
Parallel
Latency
10%
Server
20%
Parallel
10%
• Squeeze out 2 effects at the same time
– Bandwidth = 60%
– Latency = 10%
– Server = 20%
– Parallel Effects = 10%
T=0
10
20
30
40
50
60
70
80
90
100
Client
Thread A
Server
T=0
10
20
test
30
40
50
60
70
80
90
100
Client
Thread B
Server
241
Inc.
120
2/10/2011
• Remove the Server and Latency together
– Server by itself is 20%, Latency by itself is 10%
– The pair is responsible for 35 seconds of response time (35%)
– Parallel interaction of the 2 provides a bonus benefit if changed simultaneously
T=0
10
20
30
40
50
60
70
80
90
100
Client
Thread A
Server
T=0
10
20
test
30
40
50
60
70
80
90
100
Client
Thread B
Server
242
• Remove Bandwidth and Latency together
– Bandwidth by itself is 60%, Latency by itself is 10%
– The pair is responsible for 75 seconds of response time (75%)
– Parallel interaction of the 2 provides a bonus benefit if changed simultaneously
T=0
10
20
30
40
50
60
70
80
90
100
Client
Thread A
Server
T=0
10
20
test
30
40
50
60
70
80
90
100
Client
Thread B
Server
243
Inc.
121
2/10/2011
Parallel Effects Drill-Down
•
•
•
•
Can show the individual effects with their combined parallel effects together
Solid bars are the individual effects
Striped bars are the parallel effects
Optimizing both components simultaneously will give the benefits of the individual effects
AND the parallel effects
T=0
10
20
30
40
50
60
70
Parallel Effects of Bandwidth and Latency
80
90
60%
Latency
10%
Server
20%
Parallel
10%
100
Response
Time
Parallel Effects of Server
test and Latency
244
Bandwidth
Parallel Effects Summary
• Delays that can not be removed by a simple change of one parameter
• To reduce, you must change at least two parameters simultaneously
• These delays
y are not bad
– Application is efficient
– Application is doing multiple things at the same time
test
Bandw idth
Latency
Server
Parallel
245
Inc.
122
2/10/2011
User Think Time
test
Capturing a Good Transaction (Task)
• Important to get an “atomic” transaction
– Atomic transaction is the smallest transaction that you can capture effectively
– Capture should be from edge of user input to screen refresh
– Eliminate “PEBCAK”
test
247
Inc.
123
2/10/2011
Atomic Transaction Example
• Modify Customer Record
– Login
• Call up the Login screen
• Type information
• “Submit”
– Look up record
• Call up correct screen/menu
• “Submit”
– Modify data
• Mouse/cursor around
test
• “Submit”
– Logout
• Call up Logout screen
• “Submit”
248
• End user complains
“modifying a customer
record is slow”
• Need to find out what is
involved in transaction
Red arrows indicate the key part
of the transaction to capture
Identifying and Removing
User Think Time
• Always get as close as you can to a perfect “Atomic” transaction as described
in the Capture
p
Section
• However, sometimes you just can’t get the perfect capture
• “User think time” enters the trace
• AppTransaction Xpert deals with think time
– Auto detects when calling up AppDoctor
– Allows user to explicitly specify think time
test
249
Inc.
124
2/10/2011
Removing User Think Time
AppTransaction Xpert automatically identifies that the trace contains user think
time. This dialog comes up when AppDoctor is selected
also allows the user to
test
pick any dependency on
the client (first talker)
and categorize it as user
think time
250
Removing User Think Time
Original Analysis
test
User Think Time
Removed
251
Inc.
125
2/10/2011
LAB 3
LAB 3
Understanding
Parallel Effects
test
252
Lab 3: Understanding Parallel Effects
• Objectives
– Explore an application transaction that has significant parallel effects
• Trace is fairly simple
• Trace is somewhat artificial
– Practice visualizing the squeezing method to understand the behavior
– Use Drill-down and QuickPredict to understand the interactions
test
253
Inc.
126
2/10/2011
Lab 3: Conclusion
• Parallel delays may seem counterintuitive at first
• It is a real effect and does happen in the real world
• This example
p is exaggerated
gg
to make it easyy to visualize
test
254
Agenda
 Overview of TCP
- TCP Basics
- TCP Windowing
- Lab 4: Examining TCP Behavior
 Troubleshooting
T
bl h ti TCP
- Multi-User QuickPredict
- Bandwidth-Delay Product
- Lab 5: Troubleshooting a Slow File Transfer
- Detecting Packet Loss
- Frozen Window
- Nagle
Nagle’ss Algorithm
test
 Discrete Event Simulation
Instructor Demo: Nagle’s Algorithm using DES
 References
- What’s New ?
- Wrap up
255
Inc.
127
2/10/2011
TCP Basics
test
What is TCP?
• Transmission Control Protocol
• Transport
p layer
y protocol
p
Application
Application
• Layered between IP and application layer
TCP
• Typical applications such as
– HTTP (Web)
– POP, IMAP, SMTP (e-mail)
– Telnet, FTP, and more
test
257
IP
Data Link
Inc.
128
2/10/2011
Why Does TCP Exist?
1) TCP offers reliable data transport to applications
•
User data is guaranteed to be delivered
2) TCP makes applications behave in a network-friendly manner
• TCP may throttle the rate at which it sends user data
Both impact application performance
test
258
Which Applications Don’t Use TCP?
• TCP adds overhead
• TCP limits throughput
• Not all applications
pp
need what TCP provides
p
– VoIP – Voice over IP
– Video Conferencing
– SNMP – Simple Network Management Protocol
– Network Gaming Protocols
– Domain Name System (DNS)
– Network Time Protocol (NTP)
test
259
Inc.
129
2/10/2011
Motivation – Why Is TCP Important?
Example TCP/IP Stack
Application Developers
Network Managers
SOAP
SOAP
HTTP
HTTP
SSL
SSL
TCP
TCP
IP
IP
IP
ETHERNET
FIBER
ETHERNET
test
• TCP is in a key location of networked applications’ protocol stacks
– Frequently misunderstood
– This makes people and software that understand TCP valuable
260
Review: ACE Application Message Chart
Timeline
Tiers
Application
message
arrows
Application
message
groups
g
p
test
Packet
sizes
Red packets have no payload (usually acknowledgements, SYNs, or FINs)
261
Inc.
130
2/10/2011
How to Read TCP Decodes
Special flags
Size is size of entire packet
(including IP, MAC, etc.)
Sequence
number
Receive window
Destination port
Frame number
F
b ttaken
k ffrom
trace file(s). Maytest
not always
be in strict order due to trace
merge
Source port
Payload length
Acknowledgement
number
262
Opening a Connection
• Three-way handshake to open connection
• Used to exchange
– Initial SEQ numbers
– TCP Options
Cli
Client
3
C
1
A
test
2
B
Server
263
Inc.
131
2/10/2011
Closing a Connection
• Four-way handshake to close connection
• “I have no more data to send.”
• Typical example
connection closed at client
client
273
B
274
C
test
275
D
272
A
connection closed at server
264
server
Closing a Connection (cont.)
• Often only three segments used
• Second FIN also is an acknowledgement for first FIN (B&C in one packet)
• TCP Resets are an
alternate way to
terminate a connection
86
A
88
D
87
B&C
test
265
Inc.
132
2/10/2011
SEQ and ACK Numbers
• Ensures that data was received
• Sending TCP gives each segment a sequence number
– SEQ of segment K is SEQ of segment K-1 plus length of segment K-1
• Receiving TCP acknowledges reception by replying with ACK
– ACK number is the SEQ number of the next expected byte
• SEQ and ACK numbers are in
bytes, not packets
Receiver
Sender
SEQ=100
Q
LEN=10
K-1
K
test
ACK= 110
?
Seq # K-1
266
Seq # K
SEQ = 110
? LEN=55
SEQ and ACK Quiz
• Which frame(s) is 36 ACKing?
• Which frame(s) is 44 ACKing?
test
267
Inc.
133
2/10/2011
TCP Acknowledgements
• Receiver normally ACKs every other packet
– Inefficient to ACK every packet
• What if only one packet is received?
Data
test
268
Acknowledgeement
– Delayed-ACK timer must expire
– 100-200 ms (OS determines exact delay)
– Usually does not impact total application performance
Review: TCP Basics
• TCP is a connection-based protocol
• It uses a three-way handshake for opening connections
• Each side closes the connection separately
p
y
• SEQ and ACK numbers ensure reliable delivery
test
269
Inc.
134
2/10/2011
TCP Windowing
test
TCP Windowing
• TCP window size is the maximum amount of data TCP will allow
outstanding (“in-flight”)
• If window size is less than or equal to outstanding data, TCP will not send
data
• In trace below, server is waiting for ACKs before sending more data
Client
test
Server
271
Inc.
135
2/10/2011
TCP Windowing
• TCP uses the minimum size of the three windows
– Congestion control window
• Changes during connection
• Internal to sending TCP
• Not visible in real world
• Lives in the “mind” of the sender
– Receive buffer
• May change during connection
• User-configurable
• Visible as WIN= in decodes
– Send buffer
test
• Controlled by sending
application
• Not visible in decodes
• Rarely the bottleneck
272
Congestion Control Window
Receive Buffer
In-Flight Data
TCP Congestion Control Window and
Slow Start
• Estimate of how much data the network can handle
• Sendingg TCP increases the congestion
g
control window each time it receives
an ACK
– Congestion control window increases exponentially every round-trip time
• Actually stored in bytes: 1460, 2920, 5840, 11680
• When TCP first starts transmitting data, it goes slowly
– Sender starts by sending one segment, then waits for ACK
test
273
Inc.
136
2/10/2011
TCP Congestion Control Window in Slow Start
Sender
Receiver
CWND = 1
Data (S1)
ACK (S2)
CWND = 2
Data (S2)
Data (S3)
ACK (S4)
CWND = 3
Data (S4)
Data (S5)
Data (S6)
CWND = 4
test
CWND = 7
274
Data (S7)
Data (S8)
ACK
(S8)
Data (S9)
Data (S10)
Data (S10)
(S11)
ACK
Data (S10)
ACK (S12)
Bulk Data Transfer
• Congestion window is greater than receive window
• TCP sends data as soon as it receives acknowledgements
• In-flight
g data is constant
• In-flight data = outstanding data = unacknowledged bytes
• Data transfer limited by receive window
test
When you see a pattern like this, blame TCP
275
Inc.
137
2/10/2011
Review: TCP Windowing
• TCP limits the amount of outstanding data
• Minimum of three windows is used
– Send window: How much data the sender can handle
– Receive window: How much data the receiver can handle
– Congestion control window: How much data the network can handle
• Congestion window
– Initially small (1, 2, or 4 segments)
– Grows exponentially during slow start
test
276
LAB 4
LAB 4
Examining
TCP Behavior
test
277
Inc.
138
2/10/2011
Lab 4: Examining TCP Behavior
• Use AppTransaction Xpert to examine TCP behavior
• You will see
– Connection opens
– Slow start
– Congestion control window
• Raise your hand, and a TA will assist you with any questions
test
278
Lab 4: Conclusion
• TCP uses SEQ and ACK numbers to ensure reliable delivery
• Minimum of three windows (congestion, receive, and send) determines
whether TCP sends more data
• Comparison of bulk and chatty applications
– Window size and bandwidth determine throughput for bulk applications
– Latency determines throughput for chatty applications
test
279
Inc.
139
2/10/2011
When to Suspect TCP as a Bottleneck
• You should suspect TCP of causing problems when:
– You cannot explain why an application is running slowly
– The servers and network are adequately provisioned
• Servers have low load
• Sufficient bandwidth
• Links not congested
– The ACE Live™ Response Time Composition Chart shows
• Large Payload Transfer (Xfer) Times
• Large Retransmission Delays
– AppTransaction Xpert ’s AppDoctor Summary of Delays tab shows
• Large Network Transfer Effects
test
• Large Protocol delays
– AppTransaction Xpert ’s AppDoctor Diagnoses point to
• One or more TCP-related bottlenecks or potential bottlenecks
280
Troubleshooting TCP with ACE Live
- Excellent for focusing troubleshooting, monitoring TCP health
• Where to start looking, where and when to capture
• Measures connection resets, packet loss, long response times, and more
 ACE Live Response Time Composition Chart
– Large Retransmission Delay
• May be sign of packet loss
• May also be unfiltered duplicate packets
– Large Payload Transfer Times
• May be TCP-related
• May also indicate bandwidth, congestion or even server processing causes
test
281
Inc.
140
2/10/2011
Troubleshooting TCP with AppTransaction Xpert
• AppDoctor provides quick way to check for TCP problems
• Suspect TCP if the Summary of Delays shows
– Large Network Transfer Effects
– Large Protocol delays
• Use diagnosis tab to identify which
specific TCP-related issue affects
your trace
– Bottlenecks
– Potential bottlenecks
test
282
Understanding Network Transfer Effects
• Combination of three effects related to transferring data
– Bandwidth, Congestion, Protocol
• Bottleneck bandwidth is needed to separate these phenomena
– Refine Network Effects key to identification of delays due to the TCP protocol
test
283
Inc.
141
2/10/2011
How to Identify TCP Problems
• Look at the AppDoctor Diagnosis
tab
– Effect of Protocol
– Effect of Network Transfer
– Last 7 stats deal with TCP
– Helps target the specific
TCP issue
• Look at decodes for long
application transactions
• Look in the network
test and
chart at the start
end of long delays
284
How to Identify TCP Problems (cont.)
• Double-click in the Summary of Delays for more information
• Use separate drilldown screens for Protocol and Congestion
test
285
Inc.
142
2/10/2011
How to Identify TCP Problems (cont.)
• Congested networks also can cause protocol delays
• TCP implements congestion avoidance
• Single-sided captures of receiving data may make congestion look like
protocol delay or tier processing
test
286
Review: Analyzing TCP Behavior
• Look for TCP problems in traces with large amount of protocol delay
• Congestion delays also induce protocol delays
• AppDoctor
pp
(forensic)
(
)
– Separately diagnoses protocol and congestion
– Diagnoses several TCP problems
• QuickPredict (what if?)
– Separately models protocol and congestion
– Can simulate changing TCP Window Size
test
287
Inc.
143
2/10/2011
Review: ACE QuickPredict
• Quick simulation engine
– Combination of analytical and discrete event simulation
– Uses analytical models of TCP behavior
• Initially shows a pristine view of network
• Allows you to study effects of the network on the application
• Outputs graphs of response time vs. one of the following:
– Bandwidth
– Latency
– Packet loss
– Link utilization
– TCP receive window size
– NEW: WAN Acceleration parameters
• QuickPredicttest
Bar Charts supports
asymmetric window sizes
288
Review: QuickPredict Input and Output
• Y axis is always response time
• X axis is a range of values for one of:
– Bandwidth, Latency, Packet Loss, Link Utilization, TCP Window size,
and WAN Acceleration (NEW!)
Change values of other variables
to see impact on response time
test
289
Inc.
144
2/10/2011
Multi-User QuickPredict
test
Multi-User QuickPredict (MUQP)
• Single-user QuickPredict considers one client only
• MUQP allows you to have more than one
– Client and client site
– Applications (either ACE traces or ACE Whiteboard traces)
– Data centers
• Output
– Response time (including
– delay components)
– Utilization
• Results
– Average test
– N-th percentile
291
Inc.
145
2/10/2011
Inputs to MUQP
• Deployed applications (defined in ACE and/or ACE Whiteboard files)
• Network topology (number of client sites, clouds, and data centers)
• Tier-pair
p pparameters ((bandwidth, latency,
y link utilization, window size, and
packet loss)
• Application deployment options (number of transactions per hour for each
client site)
test
292
Outputs of MUQP
• "Average-case" response times for applications
• "Worst-case" results for access links (utilization) and applications (total
response time and components of tier processing, latency, protocol, and
congestion delay)
test
293
Inc.
146
2/10/2011
Instructor Demo of MUQP
test
294
Bandwidth-Delay Product
test
Inc.
147
2/10/2011
• Bandwidth-delay product — Defined as amount of data (in bytes) that can be
sent before ACK is received for the first unacknowledged bit of data
– 2 x one-way latency x bandwidth of slowest link
• To “fill the pipe,” TCP window size must be equal to bandwidth-delay product
• If window is too small, TCP must wait for acknowledgment
g
before sendingg new data
• Example below
– Bandwidth:
– Latency:
– BWDP:
– Receive Window:
– Throughput limited to:
1,544 kbps = 193 KB/sec
250 ms = 0.250 sec
2 x .25 x 193 = 96.5 KB
8 KB (1/12 ideal size)
130 kbps (1/12 link speed)
test
296
Capacity = .096 s × 125 KB/s = 12 KB (8 segments)
Bandwidth Delay Product = 2 × Capacity = 24 KB (16 segments)
Bandwidth-Delay
96 ms
Sender
Receiver
test
1 Mbps
297
Inc.
148
2/10/2011
LAB 5
LAB 5
Troubleshooting
a Slow File Transfer
test
298
Lab 5: Troubleshooting a Slow File Transfer
• Overview
– Computer manufacturer moving bulk data from Chicago to Taipei has dedicated
T1 without any packet drops
– Not seeing expected response time
– You are a network engineer
– Your manager: “It must be the network”
– Contemplating (expensive) link upgrade
• Objective
– Determine the cause of slow
response time
– Propose and test a solution
test
299
Inc.
149
2/10/2011
Lab 5: Conclusion
• Simple TCP configuration change improved performance
• Over same time period:
p
– Large receive window
– Small receive window
test
• What is the downside of making this change?
300
Changing the Receive Window
• How do I change the receive window on my computer?
– Windows
• http://www.microsoft.com/technet/network/tcpip/default.mspx
• http://www.microsoft.com/technet/itsolutions/network/deploy/depovg/tcpip
2k.mspx
• Google search: windows tcp/ip implementation details
– Solaris
• http://www.sean.de/Solaris/soltune.html#water
– Linux
• http://www.speedguide.net/read_articles.php?id=121
• Default for modern operating systems is 64 KB
test
– Maximum advertised receive window value in TCP packet is 64 KB
– How do we handle today’s large capacity, high latency pipes?
• TCP Window Scaling
301
Inc.
150
2/10/2011
TCP Window Scaling
• TCP allows for WIN= field to be scaled by a power of 2
– Decodes will show WIN<<2=262140
– WIN<<x means multiply field by 2x
• This setting is exchanged in TCP options at beginning of connection
– Both sides must agree and understand this option
test
• What if a packet trace does not contain the opening of a connection?
– AppTransaction Xpert cannot detect the presence of TCP Window scaling
302
Detecting Packet Loss
test
Inc.
151
2/10/2011
TCP and Packet Drops
• Where do packets get dropped, and why?
– Media collisions
– Wireless signal interference
– Congested networks
• Endpoints must react to overloaded routers
– Drops are a form of indirect communication between routers and senders
• Routers have multiple strategies for keeping queue size in check
– Tail drop
– Random Early Detection (RED)
test
304
Packet Loss and Retransmissions
 Packet is dropped by the network and retransmitted
 Time between when it should have arrived and when it did arrive
 Controlled by TCP timers
 Counted
C
t d as P
Protocol
t
lD
Delay
l
 Retransmission also invokes congestion control
 Slows down throughput
 Counted as Protocol Delay T = 0 1 2 3 4 5 6 7 8 9 10
Client
A
B
B
test
Protocol Delay
Server
0 1 2 3 4 5 6 7 8 9 10
305
Inc.
152
2/10/2011
Drops and Retransmissions
• TCP must make sure all bytes of a stream are received
– TCP guarantees delivery by retransmitting packets presumed dropped
• Two methods by which TCP will retransmit a dropped packet
– Retransmission timeout
• Sending TCP schedules a timer for outstanding packets
• Timer is cancelled when the packet is acknowledged
• If timer expires, packet is presumed missing, re-sent
– Triple duplicate ACKs
• ACK number indicates next expected packet
• If receiving TCP does not get the expected packet, it ACKs every packet
test
• If receiving TCP does not get the expected packet, ACK number does not
increase.
• If sending TCP receives acknowledgements with the same ACK number, it
knows a packet was likely lost
306
TCP One Loss: Retransmission Timeout
Sender
Receiver
Data (S101)
Data (S102)
Data (S103)
Data (S104)
ACK ((S103))
Data (S105)
ACK (S105)
Data (S106)
Data (S107)
Data (S108)
Retransmission
timeout
ACK (S106)
ACK (S106)
test
Data (S106)
ACK (S109)
307
Inc.
153
2/10/2011
TCP One Loss: Triple Duplicate ACK
Sender
Receiver
Data (S101)
Data (S102)
Data (S103)
Data (S104)
ACK (S103)
Data (S105)
Triple duplicate
ACK and fast
retransmit
Data (S106)
ACK (S104)
Data (S107)
ACK (S104)
Data (S108)
ACK ((S104))
test
Because 104 was
not received,
Receiver does not
change ACK
number
ACK (S104)
Data (S104)
ACK (S109)
308
Cost of Loss Detection
• Retransmission timeout
– There are not enough ACKs for triple duplicate ACK
– Cost: Timer (often large) plus time to send 1 packet
– Congestion window decreases to one segment
• Triple-duplicate ACKs
– Cause fast retransmit
– Cost: Time to send 1 packet
Retransmission
Triple-duplicate ACK
test
• Notice the send times
309
Retransmission
Inc.
154
2/10/2011
Review: Detecting Packet Loss
• Two mechanisms
– Retransmission time-out (RTO)
• Costly
• Always works
– Triple-duplicate ACK
• Relatively efficient
• Does not always happen
• Chatty applications rarely “qualify”
• Food for thought
– Why acknowledge every packet during loss?
– Why not quintuple-duplicate
ACK?
test
– Why not single-duplicate ACK?
310
Frozen Window
test
Inc.
155
2/10/2011
TCP Frozen Window
• Receiver stores packets in receive buffer
• Advertised receive window “WIN=” reduced by data in buffer
• Normal applications
pp
remove data quickly
q
y
• What happens if:
– A server is handling too many clients?
– What if the server is running other applications?
– What if a another thread is hogging the CPU?
test
312
TCP Frozen Window (cont.)
• As WIN decreases, buffer fills up, sender cannot send data
• Connection becomes “frozen”
• Sender will send pprobe packets
p
to see if window size has increased
test
313
Inc.
156
2/10/2011
TCP (Almost) Frozen Windows
• As WIN decreases, sender cannot send data
• What if the WIN is less than MSS?
• Sender could send data, but often does not
• Detection of “chilly windows” new to AppTransaction Xpert 16.0
test
314
Drilling into Protocol (cont.)
test
• Frozen Windows:
– Receiving WIN goes to zero
– Sending side sends “probe” packets
315
Inc.
157
2/10/2011
Nagle’s Algorithm
test
Nagle’s Algorithm
• Nagle’s Algorithm
– TCP mechanism to reduce sending of many tiny messages
– Sender may not send data unless a full, MSS-sized segment can be sent
OR there is no unacknowledged data
• Delayed Acknowledgement
• Interaction of these two mechanisms sometimes causes problems
• Ridiculous situation where both sides wait for the other to do something
Client
test
A
B
B
Server
317
200 ms
Inc.
158
2/10/2011
Nagle’s Algorithm Example
• Client sends segment A (less than MSS)
• Client wants to send segment B, but needs to wait for ACK
• Server is waitingg for another segment
g
before sendingg ACK
Eventually, server sends ACK
Client then sends segment B
Process repeats…
Client
test
A
B
B
Server
318
200 ms
Nagle’s Algorithm Example (cont.)
• What causes the delay for this application?
test
319
Inc.
159
2/10/2011
Nagle’s Algorithm and Delayed ACK
• To eliminate delays when sending consecutive small messages:
– Rewrite the application
• Sending multiple small consecutive messages is inefficient
• Sometimes you can’t change the application
– Disable Nagle’s algorithm
• Done within the application
• Effects only the application changed: (TCP_NODELAY socket option)
• Not always possible to change an existing application
– Acknowledge every packet
• Practical real world alternative
• On Windows: http://smallvoid.com/article/winnt-nagle-algorithm.html
test
• Warning: Applies to every application on the machine!
– Interacts with congestion control window, affecting bulk applications
320
References
• TCP in OPNET
– Online documentation
– Overview PDF
• Model Library/Standard / TCP Model Usage Guide
– ACE TCP Windowing
Wi d i Bottleneck
B ttl
k online
li documentation
d
t ti
• Applications in OPNET
– Several whitepapers including SMARTE
• Help / Product Documentation / Modeling Methodologies / Methodologies and Case
Studies
• http://www.psc.edu/networking/projects/tcptune/
– Setting system wide TCP window sizes
– Listing of TCP features for each operating system
• http://www.ncsa.uiuc.edu/People/vwelch/net_perf/tcp_windows.html
– Instructions on setting TCP window sizes within application code
• http://www.sean.de/Solaris/soltune.html
test
– Tuning your TCP/IP stack on Solaris
• TCP/IP Illustrated, Volume I, Richard Stevens
321
Inc.
160
2/10/2011
Take-Away Points
• Windowing causes TCP to wait before transmitting application data
• Bulk data-transfer throughput often limited by TCP windowing
• TCP frequently
q
y does not use the full available bandwidth
• Packet loss can cause significant reduction in throughput
• QuickPredict is a fast, easy, accurate way to predict application response time
• Use DES to model and study detailed behavior
• Simulate using the proper TCP machine settings
Next step: Run homework labs for a greater understanding of TCP
test
322
What’s New ??
test
Inc.
161
2/10/2011
What’s New in AppTransaction Xpert ??
• 64-Bit Support
– Access more that 4GB of memory on 64-bit machines which allows you analyze
much larger capture files
• Windows 7 Professional (32-bit and 64-bit)
• Capture Enhancements
– Manage captures from ACE Live appliances within the Application Capture
Manager. New ACE Live tab to quickly download and preview data
– ACE Capture Agent 3.9  Ability to filter the traffic data by time range, IP
addresses, and/or IP conversations.
– Capture File Filters  Greater flexibility in specifying hosts and ports.
• Support for Riverbed Transparent Mode
test
– The AppTransaction Xpert import now supports capture files from Riverbed
accelerators running in full or port transparency mode
324
What’s New in AppTransaction Xpert (cont.)
• Streamlined Import Process
– Quickly open capture files through the user interface without specifying any information
(such as location, bandwidth, latency, etc.)
• Bandwidth,
Bandwidth Latency
Latency, and Locations can be specified as needed
– Automatic Filtering of Duplicate Packets and TSO (TCP Segmentation Offloading)
– Simplified Manual Merge Workflow
• QuickPredict Enhancements - WAN Acceleration Support
– Support for WAN optimization technologies by allowing users to predict behavior for
compression, caching, and application acceleration
• HTTP Analysis
Anal sis Enhancements
test
– Automatic detection
of HTTP on non-standard ports (thus eliminating the need to perform
redecodes or modifying config files)
– Improved analysis for packet slicing and missing packets
– HTTP-specific visualization of object response times in TreeView
325
Inc.
162
2/10/2011
References
test
Additional Documentation
• Online at:
www.OPNET.com / Support / Support Center
–Documentation
–Release Notes
• Also look at: http://userforums.opnet.com
–Info on ALL OPNET products
–Discussion Groups
–Hints
Hi & tricks
i k
test
327
Inc.
163
2/10/2011
OPNET Training Courses
• OPNET provides Instructor-led training for all of our
products.
–OPNET
OPNET Training Facilities;
• Bethesda, Cary, Dallas, Nashua, Santa Clara
–On-site at YOUR facilities
–On the Web via WebEx
• For AppTransaction Xpert :
–2-day Instructor-led-training
test
• Register for
courses online at:
www.opnet.com/Training
328
OPNET Certification Program
Become recognized as an expert user of OPNET products…
Certification covers relevant topics such as:
•
•
•
•
•
Management / Administration
Configuration
Product Selection
Product Operation
Networking Knowledge
Upon
p successful completion:
p
• Letter
test
• Certificate
• Individual certification ID
• Valid for 2 years
329
Inc.
164
2/10/2011
AppTransaction Xpert Online Training
test
330
OPNET Web Pages - Training Resource
test
331
Inc.
165
2/10/2011
OPNETWORK is the largest event of its kind,
attracting thought leaders from industry, government,
and academic communities from all over the world
world, to
advance best practices
for:
test
• Application Performance Management
• Network Engineering, Operations, and Planning
• Network R&D
RSVP at: www.opnet.com/opnetwork2010
332
332
Course Evaluation
We’ll use this information to improve our courses and teaching methods. Please
enter as much comment material as you’d like… the more info you add, the
better we’ll get!
Point your browser to:
http://eval.opnet.com
Please enter Instructor and Course info…
– Course: Introduction to AppTransaction Xpert
– Course Date:
– Instructor Name:
FREE
Stuff !!
test
Thank You in advance for your participation!
333
Inc.
166
2/10/2011
test
334
Inc.
167

Lecture Manual

Transcription

Similar documents

Internship report - School of Engineering Science

1503 Debugging Simulation Models — Advanced

Network Simulation Tools – OPNET Modeler

Viagra Apteka Szczecin — Good Prices, Wide Choice Of Medications

Introduction to OPNET Modeler

folleto open frame - Multimedia Content Management

Using SteelApp Traffic Manager with SteelCentral Web Analyzer

View PDF - TOYDIRECTORY.com

Silego Technology - Globalpress Electronics Summit