1 How to Monitor Performance Contents

1 How to Monitor Performance
Contents
1.1.
1.2.
1.3.
1.4.
1.5.
Introduction ..................................................................................................................... 1
Performance - some theory ............................................................................................. 1
Performance - basic rules ................................................................................................ 3
Recognizing some common performance problems ........................................................... 3
Monitoring, and optimizing, the performance ..................................................................... 4
1.5.1. Tools and mechanisms ......................................................................................... 4
1.5.2. Interpreting the request.log .................................................................................... 5
1.5.3. Caching ................................................................................................................ 8
1.5.4. Analyzing Search ................................................................................................ 10
1.5.5. Monitoring Performance using JVisualVM ............................................................. 10
1.5.6. Performance when loading and editing Digital Assets ............................................ 10
1.1 Introduction
CQ5 encompasses several applications, and interacts with several more.
Performance (or the lack of it) is one of the first things that your users notice, so as with any
application with a user interface, performance is of key importance. To optimize the performance of
your CQ5 WCM installation you need to monitor various attributes of the instance and its behavior.
This is primarily of interest to power user, system administrators and project managers.
1.2 Performance - some theory
The problems that cause performance issues are often difficult to track down, even when their
effects are easy to see.
A basic starting point is a good knowledge of your system when it is operating as normal. If you
don't know how your environment "looks" and "behaves" when it is performing properly, it can
be difficult to locate the problem when performance deteriorates. This means that you should
spend some time investigating your system when it is running smoothly and ensure that collecting
performance information is an ongoing task. This will provide you with a basis for comparison
should the performance suffer.
The following diagram illustrates the path that a request for CQ5 content can take - and therefore
the number of different elements which can impact the performance.
Page 1 of 11
CQ 5.2 WCM
Copyright 1993-2009 Day Management AG
How to Monitor Performance
Figure 1.1. CQ5 request - the web-chain
Performance is also a balance between Volume and Capacity:
Volume
the amount of output that is processed and delivered by the system.
Capacity
the system’s ability to deliver the volume.
This can be illustrated in various locations throughout the web-chain.
Page 2 of 11
CQ 5.2 WCM
Copyright 1993-2009 Day Management AG
How to Monitor Performance
Figure 1.2. Capacity vs. Volume
There are several functional areas which are often responsible for impacting the performance:
• Caching
• Application (your project) code
• Search functionality
1.3 Performance - basic rules
Certain rules should be kept in mind when optimizing performance:
1. Performance tuning must be part of every project.
2. Do not optimize early in the development cycle.
3. Performance is only as good as the weakest link.
4. Always think about capacity vs. volume.
5. Optimize important things first.
6. Never optimize without realistic goals.
Note
Bear in mind that the mechanism you use to measure performance will often affect
exactly what you are trying to measure. You should always try to account for these
discrepancies, and eliminate as much of their effect as possible; in particular browser
plug-ins should be de-activated wherever possible.
1.4 Recognizing some common performance problems
The following lists common performance issues which occur, together with proposals on how to
spot and counteract them.
Page 3 of 11
CQ 5.2 WCM
Copyright 1993-2009 Day Management AG
How to Monitor Performance
Table 1.1. Recognizing common performance problems
Area
Symptom(s)
To increase capacity...
To reduce volume...
Client
High client CPU usage.
Install a client CPU with
higher performance.
Simplify (HTML) layout.
Low server CPU usage.
Upgrade to a faster browser. Improve client-side cache.
Some clients fast, some
slow.
Server
Network
CPU usage low on both
servers and clients.
Remove any network
bottlenecks.
Improve/optimize the
configuration of the client
cache.
Browsing locally on the
server is (comparatively)
fast.
Increase network
bandwidth.
Reduce the "weight" of
your web pages (e.g. less
images, optimized HTML).
Cluster your web-servers.
Reduce the hits per page
(visit).
Web-server CPU usage on the webserver is high.
Use a hardware loadbalancer.
Application Server CPU usage is high.
Cluster your CQ5 instances. Search for, and eliminate,
CPU and memory hogs (use
code review, timing output,
etc).
High memory consumption.
Improve caching on all
levels.
Low response times.
Optimize templates and
components (e.g. structure,
logic).
Repository
Cache
1.5 Monitoring, and optimizing, the performance
Performance issues may stem from a number of causes that have nothing to do with your website,
including temporary slowdowns in connection speed, CPU load, and many more.
It may also impact either all your visitors, or only a subset of them.
All this information needs to be obtained, sorted and analyzed before you can optimize the
performance.
If you experience a performance issue:
• try to replicate it: with one (or preferably more) standard web-browsers, on a different client that
you know has good general performance and/or on the server itself (if possible)
• check whether anything (related to the system) has changed within an appropriate time-space,
and if any of these changes could have impacted the performance
• collect as much information as possible to compare with your knowledge of the system under
normal circumstances
1.5.1 Tools and mechanisms
The following gives a short overview of some of the tools available for monitoring performance.
Page 4 of 11
CQ 5.2 WCM
Copyright 1993-2009 Day Management AG
How to Monitor Performance
Note
Some of these will be dependent on your operating system.
Table 1.2. Tools and mechanisms for monitoring performance
Tool
Used to analyze...
Usage / More information...
request.log
Response times and
concurrency.
Interpreting the request.log.
truss/strace
Page Loads
Unix command. Include the misc.truss log level to
INFO.
Thread dumps
Observe JVM threads. Dependent on the operating system, e.g. kill -QUIT
Identify contentions,
<pid> on Unix/Linux whereas Ctrl-Break on Windows.
locks and long-runners.
System calls
Identify timing issues.
Apache Bench Identify memory leaks,
selectively analyze
response time.
Calls to System.currentTimeMillis() or
com.day.util.Timing are used to generate
timestamps from your code, or via HTML-comments.
Note: These should be implemented so that they can
be activated / deactivated as required; when a system is
running smoothly the overhead of collecting statistics will
not be needed.
For full details: http://httpd.apache.org/docs/2.0/programs/
ab.html; basic usage is: ab -k -n <requests> -c
<concurrency> <url>
Search
Analysis
Execute search queries Analyzing Search.
offline, identify response
time of query, test and
confirm result set.
JMeter
Load and functional
tests.
http://jakarta.apache.org/jmeter/
JProfiler
In-depth CPU and
memory profiling.
http://www.ej-technologies.com/
JConsole
Observe JVM metrics
and threads.
Usage: jconsole
Note: With JDK 1.6 JConsole is extensible with plug-ins;
for example, Top or TDA (Thread Dump Analyzer).
JVisualVM
Observe JVM metrics,
threads, memory and
profiling.
Monitoring Performance using JVisualVM.
Usage: jconsole
Note: With JDK 1.6 JConsole is extensible with plug-ins;
for example, Top or TDA (Thread Dump Analyzer).
truss/strace,
lsof
In depth kernel call and Unix/Linux commands.
process analysis (Unix).
Timing
Statistics
See timing statistics for To see timing statistics for page rendering
page rendering.
you can use Ctrl-Shift-U together with ?
debugClientLibs=true set in the URL.
1.5.2 Interpreting the request.log
This file registers basic information about every request made to CQ5. From this valuable
conclusions can be extracted.
Page 5 of 11
CQ 5.2 WCM
Copyright 1993-2009 Day Management AG
How to Monitor Performance
1.5.2.1 Monitoring traffic on your website
The request log registers each request made, together with the response made:
09:43:41 [66] -> GET /author/y.html HTTP/1.1
09:43:41 [66] <- 200 text/html 797ms
By totaling all the GET entries within a specific periods (e.g. over various 24 hour periods) you can
make statements about the average traffic on your website.
1.5.2.2 Monitoring response times with the CQ5 request.log
A good starting point for performance analysis is the request log.
You can find the request log at <cq-installation-dir>/crx-quickstart/logs.
The log looks as follows (the lines are shortened for simplicity):
31/Mar/2009:11:32:57
31/Mar/2009:11:32:57
31/Mar/2009:11:33:17
31/Mar/2009:11:33:17
+0200
+0200
+0200
+0200
[379]
[379]
[380]
[380]
->
<->
<-
GET
200
GET
200
/path/x HTTP/1.1
text/html 33ms
/path/y HTTP/1.1
application/json 39ms
This log has one line per request or response:
• The date at which each request or response was made.
• The number of the request, in square brackets. This number matches for the request and the
response.
• An arrow indicating whether this is a request (arrow pointing to the right) or a response (arrow to
the left).
• For requests, the line contains:
• the method (typically, GET, HEAD or POST)
• the requested page
• the protocol
• For responses, the line contains:
• the status code (200 means “success”, 404 means “page not found”)
• the MIME type
• the response time
Using small scripts, you can extract the required information from the log file and assemble the
statistics you want. From these, you can see which pages or types of pages are slow, and if the
overall performance is satisfactory.
1.5.2.3 Monitoring search response times with the CQ5 request.log
Search requests are also registered in the log file:
31/Mar/2009:11:35:34 +0200 [338] -> GET /author/playground/en/tools/search.html?
query=dilbert&size=5&dispenc=utf-8 HTTP/1.1
31/Mar/2009:11:35:34 +0200 [338] <- 200 text/html 1562ms
Page 6 of 11
CQ 5.2 WCM
Copyright 1993-2009 Day Management AG
How to Monitor Performance
So, as above, you can use scripts to extract the relevant information and build up statistics.
1.5.2.4 Monitoring the numbers and impacts of concurrent users
Again the request.log can be used to monitor concurrency and the system's reaction to it.
Tests must be made to determine how many concurrent users the system can handle before a
negative impact is seen. Again scripts can be used to extract results from the log file:
• monitor how many requests are made within a specific time span e.g. one minute
• test the effects of a specific number of users all making the same requests at (as close as
possible) the same time; e.g. 30 users clicking Save at the same time
31/Mar/2009:11:45:29 +0200 [333] -> GET /author/libs/Personalize/content/statics.close.gif
HTTP/1.1
31/Mar/2009:11:45:29 +0200 [334] -> GET /author/libs/Personalize/content/statics.detach.gif
HTTP/1.1
31/Mar/2009:11:45:30 +0200 [335] -> GET /author/libs/CFC/content/imgs/
logo.rZMNURccynWcTpCxyuBNiTCoiBMmw000.default.gif HTTP/1.1
31/Mar/2009:11:45:32 +0200 [335] <- 304 text/html 0ms
31/Mar/2009:11:45:33 +0200 [334] <- 200 image/gif 31ms
31/Mar/2009:11:45:38 +0200 [333] <- 200 image/gif 31ms
31/Mar/2009:11:45:42 +0200 [336] -> GET /author/libs/CFC/content/imgs/
logo.rZMNURccynWcTZRXunQbbQtvuuCMbRRBuWXz0000.default.gif HTTP/1.1
31/Mar/2009:11:45:43 +0200 [337] -> GET /author/titlebar_bg.gif HTTP/1.1
31/Mar/2009:11:45:43 +0200 [336] <- 304 text/html 0ms
31/Mar/2009:11:45:44 +0200 [337] <- 304 text/html 0ms
1.5.2.5 Using rlog.jar to find requests with long duration times
CQ includes various helper tools located in <cq-installation-dir>/crx-quickstart/opt/
helpers. One of these, rlog.jar, can be used to quickly sort request.log so that requests
are displayed by duration, from longest to shortest time.
The following command shows the possible arguments:
$java -jar rlog.jar
Request Log Analyzer Version 21584 Copyright 2005 Day Management AG
Usage:
java -jar rlog.jar [options] <filename>
Options:
-h
Prints this usage.
-n <maxResults> Limits output to <maxResults> lines.
-m <maxRequests> Limits input to <maxRequest> requests.
-xdev
Exclude POST request to CQDE.
For example, you can run it specifying request.log file as a parameter and show the 10 first
requests that have the longest duration:
$ java -jar ../opt/helpers/rlog.jar -n 10 request.log
*Info * Parsed 464 requests.
*Info * Time for parsing: 22ms
*Info * Time for sorting: 2ms
*Info * Total Memory: 1mb
*Info * Free Memory: 1mb
*Info * Used Memory: 0mb
-----------------------------------------------------18051ms 31/Mar/2009:11:15:34 +0200 200 GET /content/geometrixx/en/company.html text/
html
2198ms 31/Mar/2009:11:15:20 +0200 200 GET /libs/cq/widgets.js application/xjavascript
1981ms 31/Mar/2009:11:15:11 +0200 200 GET /libs/wcm/content/welcome.html text/html
Page 7 of 11
CQ 5.2 WCM
Copyright 1993-2009 Day Management AG
How to Monitor Performance
1973ms 31/Mar/2009:11:15:52 +0200 200 GET /content/campaigns/geometrixx.teasers..html
text/html
1883ms 31/Mar/2009:11:15:20 +0200 200 GET /libs/security/cq-security.js application/
x-javascript
1876ms 31/Mar/2009:11:15:20 +0200 200 GET /libs/tagging/widgets.js application/xjavascript
1869ms 31/Mar/2009:11:15:20 +0200 200 GET /libs/tagging/widgets/themes/default.js
application/x-javascript
1729ms 30/Mar/2009:16:45:56 +0200 200 GET /libs/wcm/content/welcome.html text/html;
charset=utf-8
1510ms 31/Mar/2009:11:15:34 +0200 200 GET /bin/wcm/contentfinder/asset/view.json/
content/dam?_dc=1238490934657&query=&mimeType=image&_charset_=utf-8 application/json
1462ms 30/Mar/2009:17:23:08 +0200 200 GET /libs/wcm/content/welcome.html text/html;
charset=utf-8
Note
You may need to concatenate the individual request.log files if you need to do this
operation on a large data sample.
1.5.3 Caching
The following diagram shows the different cache locations that can be used for the various content
types.
Figure 1.3. What can be cached where?
The following can act as a rough guide for target values:
Page 8 of 11
CQ 5.2 WCM
Copyright 1993-2009 Day Management AG
How to Monitor Performance
Figure 1.4. Cache vs. Uncached - maximum hits / second
Although there are many algorithms to ensure that data is retrieved from the source system when
appropriate, circumstances can arise where the data residing in a cache is out of date. Retrieving
every page individually is the only guaranteed method of ensuring your content is up-to-date, but it
is very costly in terms of response, and can indeed cause knock-on effects.
This is particularly relevant when using personalized pages, where at least some content of a
page is dependent on the user, and the account they used to login.
Figure 1.5. Cache speed vs. Data Integrity
Page 9 of 11
CQ 5.2 WCM
Copyright 1993-2009 Day Management AG
How to Monitor Performance
1.5.3.1 Optimizing your content for cache performance
Make sure you use realistic cache settings for the browser cache. If you have disabled the browser
cache for development, this may increase traffic and decrease responsiveness.
1.5.4 Analyzing Search
First steps to analyzing the search function can be made with Monitoring search response times
with the CQ5 request.log.
However, once you have determined the response time, you may need to analyze why the request
is taking the time it does, and what can be done to improve the response. Further information about
the underlying search functionality of CRX can be found at Searching in CRX.
1.5.5 Monitoring Performance using JVisualVM
Since JDK 1.6 the tool command jvisualvm is available. After you have installed JDK 1.6 you
can:
1.
Either:
a.
Start your CQ5 instance using the -jconsole option.
b.
Add the -Dcom.sun.management.jmxremote argument to the java command line
that starts your JVM.
2.
Run jvisualvm (normally found in the JDK 1.6 bin folder).
3.
From within the Local application, double-click com.day.crx.quickstart.Main:
Note
You can use this tool to generate thread dumps and memory head dumps. This
information is often requested by the technical support team.
1.5.6 Performance when loading and editing Digital Assets
Due to the large volume of data involved when loading and editing digital assets, performance can
become an issue.
Page 10 of 11
CQ 5.2 WCM
Copyright 1993-2009 Day Management AG
How to Monitor Performance
Two things affect performance here:
• CPU - multipe cores allow for smoother work when transcoding
• Hard disk - parallel RAID disks achieve the same
To improve performance you can consider the following:
• How many assets are going to be uploaded per day? A good estimate can be based on:
• The timeframe in which edits will be made (typically the length of the working day, more for
international operations).
• The average size of images uploaded (and the size of renditions generated per image) in
megabytes.
• Determine the average data rate:
80% of all edits will be made in 20% of the time, so in peak time you will have 4 times the
average data rate. This is your performance goal.
Page 11 of 11
CQ 5.2 WCM
Copyright 1993-2009 Day Management AG