Spotlight on SQL Server Getting Started Guide

Transcription

Spotlight on SQL Server Getting Started Guide
Spotlight® on SQL Server Enterprise
10.5
Getting Started Guide
© 2014 Quest Software, Inc.
ALL RIGHTS RESERVED.
This guide contains proprietary information protected by copyright. The software described in
this guide is furnished under a software license or nondisclosure agreement. This software may be
used or copied only in accordance with the terms of the applicable agreement. No part of this
guide may be reproduced or transmitted in any form or by any means, electronic or mechanical,
including photocopying and recording for any purpose other than the purchaser’s personal use
without the written permission of Quest Software, Inc.
The information in this document is provided in connection with Quest products. No license,
express or implied, by estoppel or otherwise, to any intellectual property right is granted by this
document or in connection with the sale of Quest products. EXCEPT AS SET FORTH IN
QUEST'S TERMS AND CONDITIONS AS SPECIFIED IN THE LICENSE AGREEMENT FOR
THIS PRODUCT, QUEST ASSUMES NO LIABILITY WHATSOEVER AND DISCLAIMS
ANY EXPRESS, IMPLIED OR STATUTORY WARRANTY RELATING TO ITS PRODUCTS
INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTY OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, OR NONINFRINGEMENT. IN NO EVENT SHALL QUEST BE LIABLE FOR ANY DIRECT,
INDIRECT, CONSEQUENTIAL, PUNITIVE, SPECIAL OR INCIDENTAL DAMAGES
(INCLUDING, WITHOUT LIMITATION, DAMAGES FOR LOSS OF PROFITS, BUSINESS
INTERRUPTION OR LOSS OF INFORMATION) ARISING OUT OF THE USE OR
INABILITY TO USE THIS DOCUMENT, EVEN IF QUEST HAS BEEN ADVISED OF THE
POSSIBILITY OF SUCH DAMAGES. Quest makes no representations or warranties with respect
to the accuracy or completeness of the contents of this document and reserves the right to make
changes to specifications and product descriptions at any time without notice. Quest does not
make any commitment to update the information contained in this document.
If you have any questions regarding your potential use of this material, contact:
Quest Software World Headquarters
LEGAL Dept
5 Polaris Way
Aliso Viejo, CA 92656
email: legal@quest.com
Refer to our Web site (www.quest.com) for regional and international office information.
Patents
Protected by U.S. Patent #6,901,582. Additional patents pending.
Trademarks
Quest, Quest Software, the Quest Software logo and Spotlight are trademarks and registered
trademarks of Quest Software, Inc in the United States of America and other countries. For a
complete list of Quest Software’s trademarks, please see
http://www.quest.com/legal/trademark-information.aspx. Other trademarks and registered
trademarks are property of their respective owners.
Spotlight on SQL Server 10.5
Getting Started Guide
Friday, 30 May 2014
Table of Contents
Use Spotlight on SQL Server
5
Diagnose CPU
5
Diagnose IO
6
Diagnose Blocking
8
Monitor | Playback
10
Identify Expensive SQL
12
Turn on and configure SQL Analysis
12
Real-Time Data and Expensive SQL
14
Data Over a Selected Time Frame and Expensive SQL
15
Filter Displayed and Collected Data
16
Report on Database Growth
18
Analyze Wait Statistics
20
Find Long Running SQL
21
Configure the Virtualization Overhead Guage
22
VMware - Connection Properties | Details
23
Troubleshoot Spotlight on SQL Server
24
Installation Issues
Remote installation of the diagnostic server fails
Connection Issues
Problems connecting to a SQL Server Instance
24
24
24
24
WMI Errors
25
Miscellaneous Issues
28
Monitor Multiple SQL Server Instances
28
Unable to perform certain operations
28
Spotlight on SQL Server - Missing Performance Counters
29
Spotlight on SQL Server - SQL Activity Drilldown | Sessions page
29
Spotlight on SQL Server Enterprise Getting Started Guide
4
Table of Contents
Playback
30
Appendix: Contact Quest
31
Contact Quest Support
31
Spotlight Support
31
Contact Quest Software
32
About Quest Software
32
Index
33
1
Use Spotlight on SQL Server
Diagnose CPU
Spotlight continually monitors CPU usage on the target server and alarms if it crosses a
threshold. The thresholds are:
CPU Usage
Alarm severity
0 – 67%
Normal
67 – 80%
Low
80 – 93%
Medium
93 – 100%
High
A CPU alarm indicates that a bottleneck on CPU is likely. This bottleneck could exist for a
number of reasons – expensive queries could be running, excessive compilation or recompilation
could be occurring, or a process on the server, not even related to SQL Server, is ‘hogging’ the
CPU. With Spotlight, you can investigate the causes of a bottleneck both in real-time and in the
immediate past.
When Spotlight detects that there is a bottleneck on the CPU, an alarm is raised. You can see
alarms on the Spotlight home page, and in Spotlight Today.
Spotlight on SQL Server Enterprise Getting Started Guide
6
Use Spotlight on SQL Server
Our first order of business is to find out what process is using all of the CPU.
From the Spotlight Home Page
From Spotlight Today
Click Diagnostics.
Select the alarm and click Diagnose.
This opens the Diagnostics drilldown. The Diagnostics drilldown helps you answer
questions such as:
l
Why is CPU usage higher than usual?
l
Is the SQL Server causing high CPU usage?
l
If it is the SQL Server, is it optimizing queries or executing them?
The first grid on the CPU page shows possible causes of CPU pressure and the likelihood that
each is contributing to high CPU usage on the server being monitored. Click an item in the grid
to display an explanation of the diagnosis and supporting information.
Diagnose IO
Sustained high I/O wait time rates are a good indicator that of a disk subsystem bottleneck and
that I/O device service times will be degraded. When Spotlight detects unusually high disk
activity on the SQL Server machine it raises an alarm.
Spotlight looks at SQL Server I/O wait times to determine if there is an IO bottleneck. A value
larger than 15 ms/IO indicates a possible bottleneck.
The thresholds for the I/O wait time alarm are:
Spotlight on SQL Server Enterprise Getting Started Guide
7
Use Spotlight on SQL Server
I/O Wait Time
Alarm severity
0 - 15
Normal
15 - 50
Medium
50 - infinity
High
After detecting a potential disk bottleneck, Spotlight displays an alarm on the home page and in
Spotlight Today.
The first thing we need to do is identify the process that is generating high amounts of
I/O activity.
From the Spotlight Home Page
From Spotlight Today
Click Diagnostics | IO.
Select the alarm and click Diagnose.
This opens the I/O Diagnostics drilldown. The I/O Diagnostics drilldown helps you answer
questions such as:
l
Why is the I/O higher than usual?
l
Is the high I/O due to paging activity?
l
Is the SQL Server causing the high I/O?
l
If the SQL Server is causing the high I/O, is it through query executions, scan operators,
PAGE_IO_LATCH or LOG waits?
Spotlight on SQL Server Enterprise Getting Started Guide
8
Use Spotlight on SQL Server
The first grid on the I/O page shows the possible causes of high I/O and the likelihood of each
contributing to this instance of the alarm. Click an item in the grid to display an explanation of
the diagnosis and supporting information.
Diagnose Blocking
Blocking in SQL Server occurs when a session that was performing some task is unable to
progress because it must wait on a resource that is currently being used by another session. The
resource that is being waited on can be either a physical structure like a lock on a table or an
internal SQL Server resource such as a latch.
Excessive blocking can be a major cause of poor application performance since it reduces the
throughput of the system. Often a user of an application does not realize that they are waiting on
a resource held by another user. From their point of view, it often seems like their application has
stopped responding.
When diagnosing blocking, you want to start your investigation by answering the
following questions:
l
Who is waiting on what?
l
How long have they been waiting?
l
What SQL was running while they were waiting?
Spotlight not only alerts you to blocking issues in your database but also helps you answer the
above questions so you can quickly resolve the problem causing the blocking.
Spotlight on SQL Server Enterprise Getting Started Guide
9
Use Spotlight on SQL Server
When Spotlight detects a blocking issue, an alarm is displayed on the home page.
The first thing we want to do is look at the Blocking drilldown. The Blocking drilldown shows
details about the current blocks and in particular, answers the questions stated above.
On the Spotlight homepage, click the Blocked Processes button, and then click the
“Blocking” drilldown.
The Blocking grid shows the sessions that are either blocked or are causing other sessions to be
blocked. The hierarchy in the tree diagram makes it easy to see which session is causing the
blocking. In addition, the resource on which the sessions are waiting is displayed. You can ask
Spotlight to resolve these to actual object names by clicking on the ‘?’ button. The time spent
waiting and the SQL last executed are also displayed.
Spotlight on SQL Server Enterprise Getting Started Guide
10
Use Spotlight on SQL Server
From here, you can view session details.
1. Select the entry in the grid.
2. Click View Session Details.
Monitor | Playback
You can view events and data collected in the recent past as though they were happening in
real-time. You can do this via the Playback ribbon.
To playback time
1. Select a connection from the Spotlight browser.
2. Select the Monitor ribbon tab.
3. Click to show:
Alarms
Description
Playback
Select an alarm or moment in time in the past to return to. Once the
time is selected, the Spotlight home page, Spotlight Today and
page drilldowns show details from that time identical in format to
that displayed in "live" state, so you can view historical data in the
same way as you view a live connection.
Tip: Click Real Time to return to the present time.
Note: Playback / Historical data is not displayed for SQL Azure
database information.
Spotlight on SQL Server Enterprise Getting Started Guide
11
Use Spotlight on SQL Server
Alarms
Description
Control
Description
Select the date. Use the back and forward buttons
to change the date.
Click the time to playback to.
The alarms list scrolls to show those alarms
occurring at any given time. Select to see the
behavior at the time when an important alarm was
raised.
Note: The time scale is colored according to the
most severe alarm for the connection.
Tip: Press and hold to magnify the time scale.
The alarms occurring on the selected date.
Click an alarm to playback to the time the alarm
occurred.
Note: The color indicates the severity of the alarm.
Real
Return to present time the Spotlight home page and page
drilldowns.
Time
Rewind
Go back in time on the Spotlight home page or page drilldown.
Click the associate arrow to define how far back to travel: 1 minute,
5 minutes, 10 minutes, 1 hour or 1 day.
Skip
Starting from the past, skip forward in time on the Spotlight home
page or page drilldown. Click the associate arrow to define how far
forward to skip: 1 minute, 5 minutes, 10 minutes, 1 hour or 1 day.
Spotlight on SQL Server Enterprise Getting Started Guide
12
Use Spotlight on SQL Server
Alarms
Description
Play
Starting from the past, step forward in time through the alarms in
sequence. Show on the Spotlight home page or page drilldown.
Click the associate arrow to define the speed of play.
Tip: On the Spotlight Home Page you can view the recent history of a single component.
Identify Expensive SQL
Identifying expensive SQL can be a time consuming process. Use the SQL Analysis feature in
Spotlight to identify SQL that is consuming resources on your server.
With SQL Analysis, you can quickly answer questions such as:
l
What SQL has executed the most?
l
What SQL consumed the most CPU or I/O in total?
l
What SQL consumed the most Average CPU or I/O?
l
What SQL took to longest time to execute?
Answering these questions will help you to direct your tuning efforts more productively.
SQL Analysis uses either ROWSET or sampling Server Side trace to collect raw data on SQL
statement execution. It then aggregates this data by similar SQL statements so that potentially
thousands or even millions of rows of raw data are distilled down into an aggregate picture
of workload.
You can set filters at point of collection so that only the events that you are interested in are
collected and when viewing the data, you can also filter the data to narrow the focus down
even further.
Turn on and configure SQL Analysis
1. Click Configure | SQL Analysis
2. Choose one of the following from the connection list:
l
l
To change the setting for an individual connection, select the connection name.
l
Select Override the default settings.
l
Select Enable SQL Analysis.
To change the setting for all connections, select Default Settings and select
Override the default settings.
Spotlight on SQL Server Enterprise Getting Started Guide
13
Use Spotlight on SQL Server
3. Click Data Collection.
4. Configure the following fields on the Data Collection page:
Option
Description
SQL Analysis is
scheduled to collect
every n minutes
Note: This feature is available only in Spotlight on SQL
Server Enterprise.
How often Spotlight collects SQL Analysis data. The
default is six hours.
To change how often Spotlight collects SQL Analysis
data
a. Click the link.
b. Select Override the default settings for the
collection 'SQL Analysis'.
c. Under ‘Store reporting data in the Spotlight
Statistics Repository’, click the Store data link.
d. Set the collection schedule. Data can be collected
at regular intervals or at a specific time on one or
more days.
Use continuous
Rowset trace
Select to use Rowset trace to collect SQL Analysis data.
Use sampling Server
Side trace
Select to use sampling Server Side trace to collect SQL
Analysis data.
Rowset trace is convenient though may affect throughput
on busy servers. It should be used with care.
Server Side trace is more suitable for busy servers than
Rowset trace. Server Side trace requires additional
configuration.
Run trace for or until
trace file reaches
Server Side trace will run until one of these conditions is
met.
Specify the maximum amount of time Server Side trace
should run for.
Specify the maximum size of the trace file.
Server trace file
location (on the SQL
Server host)
Type a location for the server trace file. The path is
relative to the SQL Server instance.
Retrieve data through
SQL Server
Select to process trace files on the SQL Server instance.
This option has the potential to affect throughput on busy
servers.
For example, C:\Trace Files.
Spotlight on SQL Server Enterprise Getting Started Guide
14
Use Spotlight on SQL Server
Option
Description
Use this option when there are permission or firewall
settings preventing the Spotlight Diagnostic Server
machine from retrieving files from the SQL Server
machine.
This is the default option.
Retrieve data from the
file system
Select to process trace files on the Spotlight Diagnostic
Server. The Spotlight Diagnostic Server machine must
have the appropriate permissions required to connect to
the SQL Server instance.
This option is not available when configuring default
settings.
Trace file location
(from Diagnostic
Server host)
Type the location of the server trace files on the SQL
Server instance. The location should be a shared folder the
Spotlight Diagnostic Server can connect to.
For example, \\computername\Trace Files.
This option is not available when configuring default
settings.
Real-Time Data and Expensive SQL
You can use the SQL Analysis grid when load testing in a testing environment. It shows SQL
Analysis data in real time.
If you are trying to answer the question “I am simulating application workload in my test
environment and I want to look at a breakdown of SQL statements in real-time. I am interested in
what SQL statements are consuming the most average CPU right now.”, you would want to look
at an aggregate of SQL statement executions (workload) in real-time. The SQL Analysis grid in
the SQL Activity drilldown allows you to do exactly that.
1. Select a SQL Server connection from the Spotlight browser.
2. Click Monitor | SQL Activity.
3. Select SQL Analysis.
4. Do the following:
To identify
Sort the grid by
SQL that consumed the most CPU
Total CPU
Spotlight on SQL Server Enterprise Getting Started Guide
15
Use Spotlight on SQL Server
To identify
Sort the grid by
The average CPU consumed by a particular
statement
Average CPU
SQL that was executed the most
Execution Count
SQL with the highest logical I/O
Average Reads
Tips:
l
Use filters to refine the data shown in the grid. See "Filter Displayed and Collected Data"
(page 16) for more information.
l
Some columns are hidden by default. To view hidden columns, right-click the grid header
row and select Organize Columns.
l
Results are cleared when you leave the SQL Analysis grid. Click Clear Grid to clear the
grid manually.
l
To wrap a column onto multiple lines, right-click the data content of the grid and select
Properties | Options.
Note: Word wrap may degrade the performance of Spotlight.
l
If Quest SQL Optimizer is installed, you can use it to tune the non-conforming SQL that
has been identified. To do this, click Optimize SQL.
Data Over a Selected Time Frame and Expensive SQL
Note: This feature is available only in Spotlight on SQL Server Enterprise.
The SQL Analysis - Workload view shows the aggregate workload over a selected time period.
Use the SQL Analysis - Workload view when diagnosing slow response times.
If you are trying to answer the question “What SQL statement consumed the most CPU over
month end processing?”, you would want to look at an aggregate of SQL statement executions
(workload) over a time range. The SQL Analysis – Workload view allows you to do exactly that.
1. Select Reporting and Trending | Views | SQL Server Performance | SQL Analysis
Workload in the Spotlight Browser.
Spotlight on SQL Server Enterprise Getting Started Guide
16
Use Spotlight on SQL Server
2. Click Report | Select Connections and select at least one connection.
3. Select a Time Range.
4. Click Refresh.
5. Do the following:
To identify
Sort the grid by
SQL that consumed the most CPU
Total CPU
The average CPU consumed by a particular
statement
Avg CPU
SQL that was executed the most
Execution Count
SQL with the highest logical I/O
Avg Reads
Tip: Use filters to refine the data shown in the grid. See "Filter Displayed and Collected
Data" (page 16) for more information.
Filter Displayed and Collected Data
To filter the data displayed
Start from the data displayed in the SQL Analysis grid or the SQL Analysis Workload view
Spotlight on SQL Server Enterprise Getting Started Guide
17
Use Spotlight on SQL Server
SQL Analysis grid
SQL Analysis Workload view
1. Click Change Filter.
1. Click Filter Data.
2. Select Filter results.
2. Select Filter results.
To add a filter to the list
1. Click Add.
2. From the Column list, select the column you want to base your filter on.
3. From the Condition list, select a condition to apply to the column.
4. In the Value field, type a value to filter events by. You can use the percent sign (%)
wildcard with the is like/is not like condition.
Tips:
l
You cannot use wildcard characters for the Database name when connected to a SQL
Server 2000 instance.
l
To turn off filtering, clear the Filter results checkbox.
l
To edit or delete a filter select it in the filter list and click the appropriate button.
To filter the data collected
1. Click Configure | SQL Analysis.
2. Choose one of the following from the connection list:
l
To change the setting for an individual connection, select the connection name.
l
Select Override the default settings.
Select Enable SQL Analysis.
l
To change the setting for all connections, select Default Settings and select
Override the default settings.
3. Ensure the Filters page is selected.
l
Click Add.
Tips:
Spotlight on SQL Server Enterprise Getting Started Guide
18
Use Spotlight on SQL Server
l
You can use the percent sign (%) and underscore (_) as wildcards.
l
You cannot use wildcard characters in the Database name field when
connected to a SQL Server 2000 instance.
l
When filtering on a string field such as DatabaseName or
ApplicationName, use only the "LIKE" and "NOT LIKE" operators. The
other operators apply only to numeric values.
Tips:
l
Use the Arrow buttons to change the order of the filters.
l
Edit or delete a filter by selecting it in the filter list and clicking the
appropriate button.
l
Filtering affects the average, minimum, maximum, and last values displayed in
the grid, as calculations are performed only on data retrieved from the SQL
Server instance.
l
To turn off filtering, on the Filters page, click Clear. This removes all filters.
4. You can specify the number and type of SQL Executions retrieved from the SQL
Server instance
a. Select the Advanced page.
b. In the Aggregate the top n SQL statements field, enter the maximum number of
SQL executions to be retrieved from the SQL Server instance.
c. From the Sorting by list, specify the criterion you want Spotlight to use to
determine what the "top" SQL statements are. Choose from Average Duration,
Average CPU, or Average IO.
Spotlight aggregates the data collected and then stores only the aggregated data.
Report on Database Growth
Note: This feature is available only in Spotlight on SQL Server Enterprise.
To report on database growth across multiple SQL servers, use the Large Databases view in
Spotlight Reporting and Trending. The Large Databases view shows the largest databases on the
selected SQL Servers. With Spotlight, you can view this data in a number of different ways.
1. Select Reporting and Trending | Views | SQL Server Environment | Large Databases
in the Spotlight Browser.
Spotlight on SQL Server Enterprise Getting Started Guide
19
Use Spotlight on SQL Server
2. Click Report | Select Connections and select at least one connection.
3. Select a Time Range.
4. Click Refresh.
By default, the data is shown as a bar chart, listing each database on the selected servers in order
of size. You can change the way the data is displayed using the Chart View button, located
above the chart.
For example, if you wanted to view the growth of your largest databases over time you can
change the chart to a line graph. To do this:
Spotlight on SQL Server Enterprise Getting Started Guide
20
Use Spotlight on SQL Server
1. Click the Chart View button and then select Line from the list.
2. Click Refresh.
To look at the space consumption proportionally you could use a pie chart. Click
Chart View | Pie.
You can use the data and chart from the view in your own reports by exporting the view. To do
this, click Export and choose a location to save the files. Spotlight saves the chart as a JPG file
and the grid as a CSV file.
Analyze Wait Statistics
When troubleshooting SQL Server performance issues, one area you can focus your investigation
on is waits. Long wait times can be an indicator of performance bottlenecks so identifying where
waits are occurring may assist you in your tuning efforts.
Spotlight not only provides an easy way to see if your SQL server has a problem with waits but
also identifies on which particular resource waits are occurring. Spotlight takes the raw data from
SQL Server and automatically calculates the rate of wait over time so you have a more current
perspective.
Spotlight on SQL Server Enterprise Getting Started Guide
21
Use Spotlight on SQL Server
To start your investigation into waits, go to the SQL Activity drilldown and click the Wait
Statistics tab.
The Waits Detail grid shows detailed waits statistics for the SQL Server instance being
monitored. It shows all wait types in SQL Server and breaks down the statistics into signal time
and resource wait time. (SQL Server provides only wait time and signal wait time.) By default,
the Waits Detail grid is sorted by Wait Time Rate (ms/s) which allows you to immediately see
which wait types are being waited on right now.
Find Long Running SQL
The SQL - Long Running SQL alarm alerts you to SQL that has been running longer than a
specified time frame. Because this time frame is unique to each environment, the SQL - Long
Running SQL alarm, by default, is not configured.
Scenario
You work in an OLTP environment and often get complaints about
delays in server processing. You want to be notified when SQL on a
particular server is running longer than expected so you can investigate
what is causing the delay. You configure Spotlight to raise a medium
severity alarm when it detects SQL running for longer than five minutes.
When the SQL - Long Running SQL alarm is raised, you use the
Sessions drilldown to view the long running SQL.
Spotlight on SQL Server Enterprise Getting Started Guide
22
Use Spotlight on SQL Server
To configure the Alarm: SQL - Long Running SQL
1. Click Configure | Alarms.
2. Select a connection to configure. Select Default Settings to apply the configuration to all
connections.
3. Select the alarm SQL - Long Running SQL.
4. Select Override the default settings for the alarm "SQL - Long Running SQL".
5. Click Add Severity and select a severity.
Scenario: Click Add Severity and select Medium.
6. Select the check box of the new severity.
7. Copy and paste the text in the Description cell from the Normal severity to the new
severity.
8. In the Start cell, type the duration in seconds, of how long the SQL should run for before
the alarm is raised.
Scenario: In the Start cell, type 300 .
Tip: You can also configure alarm severities using keys. In the SQL - Long Running SQL alarm,
the key is the SPID. See "Configure Keyed Alarms" in the online help for more information.
Configure the Virtualization Overhead Guage
The Virtualization Overhead gauge shows the percentage of CPU that is unavailable to this
virtual machine because it is being consumed either by other virtual machines or by
VMware itself.
In a virtual environment, the physical CPU of a host is shared by virtual machines and the
VMware Hypervisor. This means there can be multiple processes wanting to use the physical
CPU of the host at the same time. The result is that virtual machines may be ready to run but
have to wait to be scheduled on a CPU. Vmware call this “ready time”. The Virtualization
Overhead gauge shows the amount of ready time as a percentage of the theoretical maximum
CPU available to the virtual machine.
To see virtualization data on Spotlight Home Pages
1. There needs to be a connection to the VMware Server. See Connections / Recent
Connections for more information.
2. In the Spotlight Browser, right-click the Windows connection and select Properties.
3. In the Host field, select the connection to the VMware Server.
4. In the VM Name field, select the name of the virtual machine.
Spotlight on SQL Server Enterprise Getting Started Guide
23
Use Spotlight on SQL Server
The Virtualization Overhead gauge now displays data on the corresponding Windows Home
Page and SQL Server Home Page.
Tip: SQL Server connections automatically use the virtualization data from their corresponding
Windows connections. You do not need to configure the SQL Server connection once the
Windows connection is configured.
VMware - Connection Properties | Details
To create a connection or edit connection properties, click Configure | Connections.
Select VMware - Connection Properties to create a VMware connection to an ESX or
VirtualCenter server that hosts a Windows machine.
Field
Description
Address
The IP address or hostname of the ESX server or VirtualCenter server that
hosts the Windows machine.
Tip: If you have a VirtualCenter server then connect to that in preference to
the ESX host. That way if the virtual machine migrates from one host to
another the VirtualCenter server can still be interrogated for data for the
Virtual Machine.
User
A valid user account on the target system.
Note: For Spotlight to monitor a VMware server, the username specified must
have at least a read-only role for the ESX server or VirtualCenter server that
you want to connect to.
Password
The password required to connect the specified user to the server.
2
Troubleshoot Spotlight on SQL Server
Installation Issues
Remote installation of the diagnostic server fails
Note: This feature is available only in Spotlight on SQL Server Enterprise.
Remote installation of the Diagnostic Server fails if the user has both a domain account and a
local account with the same user name.
Additionally, if the user uses a domain account that does not have permission to access the
remote machine (that is, the account does not have administrator privileges) then the remote
installation will not complete.
Connection Issues
Spotlight on SQL Server retrieves its data from the SQL Server via the Spotlight Diagnostic
Server. The Spotlight Diagnostic Server also uses WMI queries to retrieve performance counter
information from the Windows server on which SQL Server is running.
The Spotlight Diagnostic Server cannot connect to a server unless it can access all of this
information. This means you need to have the appropriate permissions on the server.
Also, the data must be retrievable over your network. Spotlight on SQL Server requires more
than just a specific port TCP/IP connection to SQL Server to do this. The port on the Spotlight
Diagnostic Server must be open. For more information, see the Spotlight on SQL Server
Deployment Guide. This document is available in PDF from Start | Quest Software | Spotlight.
Problems connecting to a SQL Server Instance
If you have trouble connecting to a SQL Server instance, check that you can:
l
Connect to the SQL Server using another tool such as Microsoft's SQL Server
Management Studio, or sqlcmd.
l
Connect to WMI using another tool such as Microsoft's WMI CIM Studio, or wmic.
(Troubleshooting WMI)
If you have problems with any of these checks, you will need to investigate and resolve these
issues before you can connect to that server using Spotlight on SQL Server.
Spotlight on SQL Server Enterprise Getting Started Guide
25
Troubleshoot Spotlight on SQL Server
WMI Errors
The following is a guide to troubleshooting WMI errors.
Error
Solution
The RPC server is
unavailable.
Spotlight on SQL Server is not able to establish a connection with the target computer.
Exception from
HRESULT:
0x800706BA
Verify the following:
l
The address of the server is entered correctly via the Spotlight Connection Manager.
Verify correct host name or IP address (Configure | Connections). Verify the host is
available (not currently shut down).
l
The Remote Procedure Call (RPC) service is running on the target computer. Verity
that "Remote Procedure Call (RPC)" is running and set to auto start after restart.
l
TCP Port 135 is open to internal traffic on the Spotlight Diagnostic Server and the
target computer.
WMI opens an undetermined port in addition to port 135. This can be troublesome in a
firewalled environment. Specifying a port range for WMI is recommended for this type
of environment. See How to limit the number of ports used by WMI for more
information.
l
The target computer is not blocked by the firewall.
Either configure WMI to use a fixed ports range (How to limit the number of ports used
by WMI) or enable remote administration exception.
Follow these steps to enable remote administration exception.
On the target computer:
1) Open the Group Policy Object Editor (gpedit.msc), open Computer Configuration |
Administrative Templates | Network | Network Connections | Windows Firewall
2) Open either Domain Profile or Standard Profile, depending on which profile you
want to configure.
3) Enable the following exceptions: "Allow Remote Administration Exception" and
"Allow File and Printer Sharing Exception".
http://msdn.microsoft.com/en-us/library/aa389286(VS.85).aspx
Access is denied.
Exception from
HRESULT:
l
The "TCP/IP NetBIOS Helper" service is running. Verity that "TCP/IP NetBIOS
Helper" is running and set to auto start after restart.
l
The "Windows Management Instrumentation" service is running on the target
computer. Verity that "Windows Management Instrumentation" is running and set to
auto start after restart.
The Windows user specified to monitor the target computer (Configure | Connections) is
unknown to the target or does not have administrator rights.
See Windows Account Permissions for more information.
Spotlight on SQL Server Enterprise Getting Started Guide
26
Troubleshoot Spotlight on SQL Server
Error
Solution
0x80070005 (E_
ACCESSDENIED)
WMI connection
time outs
WMI query failed:
Invalid class.
The timeout value defaults to the value DCOM specifies (usually 60 seconds).
You can adjust this value via dcomcnfg.exe
The WMI class does not exist on the target machine
Solution: Recreate the WMI classes.
[0x80041010]
Windows Connections
Either of the following will recreate the WMI classes:
l
Open a command prompt window and run the following command:
wmiadap /f
l
Use the Microsoft Extensible Counter List (Exctrlst) utility to enable the following
counters:
Win32_PerfDisk, Win32_PerfNet, Win32_PerfOS and Win32_PerfProc.
http://www.microsoft.com/downloads/details.aspx?displaylang=en&familyid=7FF99683B7EC-4DA6-92AB-793193604BA4
Note: A reboot of the target machine is required.
SQL Server Analysis Services Connections
Either of the following will recreate the WMI classes:
l
Open a command prompt window on the machine hosting the SQL Server Analysis
Services instance. Run the following command:
wmiadap /f
l
Unregister and re-register the WMI classes. To do so:
1. On the problematic machine with SQL Server Analysis Services installed,
find out the SQL installation path. See below for the default installation
path. Your installation path may be different.
SQL Server Analysis Services 2012
C:\Program Files\Microsoft SQL
Server\MSAS11.MSSQLSERVER\OLAP\bin\Counters
SQL Server Analysis Services 2008
C:\Program Files\Microsoft SQL
Server\MSAS10.MSSQLSERVER\OLAP\bin\Counters
SQL Server Analysis Services 2005
C:\Program Files\Microsoft SQL Server\MSSQL.2\OLAP\bin
Spotlight on SQL Server Enterprise Getting Started Guide
27
Troubleshoot Spotlight on SQL Server
Error
Solution
Note: You are looking for the folder that contains the following files:
For unamed instances:
MSSQLServerOLAPService
perf-MSSQLServerOLAPServicemsmdctr.ini
For named instances:
MSOLAP$Your_SSAS_NamedInstanceName
perf-Your_SSAS_NamedInstanceNamemsmdctr.ini
If you need to check the named instance name, use services.msc.
2. At the command prompt, change folder to the installation path.
For example, cd C:\Program Files\Microsoft SQL
Server\MSAS10.MSSQLSERVER\OLAP\bin\Counters
3. Run the following commands to unload and load counters:
unamed
instances
unlodctr MSSQLServerOLAPService
named
instances
unlodctr MSOLAP$Your_SSAS_NamedInstanceName
lodctr perfMSSQLServerOLAPServicemsmdctr.ini
lodctr perf-Your_SSAS_
NamedInstanceNamemsmdctr.ini
Tip: Replace Your_SSAS_NamedInstanceName with the
SQL Server Analysis Services named instance name. If
you need to check the named instance name, use
services.msc.
4. Run the following command to parse all the performance libraries on the
system and refresh the performance counter classes on the problematic
machine:
wmiadap /f
5. Use wbemtest.exe to verify the WMI Classes exist.
WMI query failed:
Invalid query.
[0x80041017]
The columns returned by the WMI class are incomplete or unexpected.
Solution: Update the WMI classes by running the following command at the command prompt
of the target computer.
wmiadap.exe /f
See http://msdn.microsoft.com/en-us/library/aa394528(VS.85).aspx for more information.
0x80014064
User credentials
cannot be used for
local connections
This error occurs when wmic command is executed locally on the target machine and
credentials specified
Spotlight on SQL Server Enterprise Getting Started Guide
28
Troubleshoot Spotlight on SQL Server
Error
Solution
[0x800705af]
The paging file is too small for this operation to complete. You may find the following link
useful:
http://blogs.technet.com/b/kevinholman/archive/2010/06/09/wmi-leaks-memory-on-server-2008r2-monitored-agents.aspx
WMI query failed:
Out of memory.
[0x80041006]
1. At the command prompt run "wbemtest"
2. Connect to the "root" namespace (not "root\default", just "root")
3. Click Open Instance. Specify "__ProviderHostQuotaConfiguration=@"
4. Select Local Only for easier readability. You will see the threshold values.
5. Increase the MemoryPerHost value to something greater. For example, double it (256
MB)
6. Save Property
7. Save Object
8. Click Exit.
9. Restart WMI services.
Invalid verb
The wmic command has attempted to access a WMI class that does not exist.
Solution: Check the spelling of parameters on the wmic command.
Invalid Global
Switch
The specified host, user or domain name contains special characters like '-' or '/'.
Solution: Modify the command by adding quotation marks.
wmic /node: 'MonHostFQDN' /user: 'DOMAIN\USER' path Win32_PerfRawData_
PerfDisk_LogicalDisk get FreeMegabytes
Miscellaneous Issues
Monitor Multiple SQL Server Instances
Spotlight on SQL Server is designed to monitor a maximum of 100 SQL Servers, Analysis
Services or Replication instances and 100 Windows servers. Exceeding this recommended limit
on 32 bit environments may result in poor performance or product instability, due to the 1 Gb
memory limit. On 64 bit environments theoretically more connections can be monitored as the 1
Gb limit no longer applies, however testing of this has been limited.
Unable to perform certain operations
If, whilst using the Spotlight on SQL Server client, a message is displayed stating:
You are not connected to the Diagnostic Server as a member of the Spotlight Diagnostic
Administrators group. Only members of that group can perform the requested operation.
Spotlight on SQL Server Enterprise Getting Started Guide
29
Troubleshoot Spotlight on SQL Server
and the user needs to be able to perform the operation that led to this message, the Spotlight on
SQL Server client user must be added to the Spotlight Diagnostic Administrators group on the
Diagnostic Server.
See “Update the Spotlight Diagnostic user groups” in the Spotlight on SQL Server Deployment
Guide, for more information.
Spotlight on SQL Server - Missing Performance Counters
Spotlight on SQL Server uses the SQL Server virtual table sysperfinfo (SQL Server 2000) and
sys.dm_os_performance_counters (SQL Server 2005 or later) to retrieve data for many of its
displays. In some rare cases, this table may not contain information. When this is the case,
Spotlight on SQL Server will not be able to collect the data it requires, and will display "0" for
many of its components. Most obvious will be the Memory icons on the home page, which will
show 0 MB of memory used by SQL Server. Also, many of the flows on the home page will
show no activity, and many drilldowns will show incomplete information.
Spotlight on SQL Server raises an informational alarm shortly after connecting if it detects that
the sysperfinfo or sys.dm_os_performance_counters table contains no data.
You can confirm the existence of this problem by running the following SQL in Query Analyzer
for SQL Server 2000:
select * from master..sysperfinfo
or by running the following SQL in SQL Server Management Studio for SQL Server
2005 or later:
select * from sys.dm_os_performance_counters
If this query returns no records, then your SQL Server performance counters are missing and
Spotlight on SQL Server will not be able to operate correctly.
Sometimes, for a variety of reasons, the SQL Server Performance Monitor counters will not show
up as they should. Often, but not always, this problem can be fixed by following these steps:
1. At the command prompt, type the following: unlodctr.exe MSSQLServer
2. Then type: lodctr.exe <SQL Server path>\binn\sqlctr.ini
3. Reboot the server.
More information can be found in the SQL Server Knowledge Base at msdn.microsoft.com.
Spotlight on SQL Server - SQL Activity Drilldown |
Sessions page
If errors are displayed on the Session Trace page (SQL Activity drilldown | Sessions page), install
the following on the Spotlight Diagnostic Server machine:
Spotlight on SQL Server Enterprise Getting Started Guide
30
Troubleshoot Spotlight on SQL Server
l
SQL Server Management Tools
The version of SQL Server Management Tools required is dependent on the latest version
of SQL Server monitored. When monitoring SQL Server 2012, SQL Server 2012
Management Tools are required.
l
Windows XP Service Pack 2
Applies only if the Spotlight Diagnostic Server machine is running on Windows XP.
Playback
Playback data is collected at scheduled intervals, in response to some alarms, and also while
viewing a drilldown using the Spotlight client. This means that history data may not be
available for a drilldown in Playback mode. You can change the rate at which data is collected.
See "Configure Scheduling" in the online help for more information.
Appendix: Contact Quest
Contact Quest Support
Quest Support is available to customers who have a trial version of a Quest product or who have
purchased a Quest product and have a valid maintenance contract. Quest Support provides
unlimited 24x7 access to our Support Portal at www.quest.com/support.
From our Support Portal, you can do the following:
l
Retrieve thousands of solutions from our Knowledge Base
l
Download the latest releases and service packs
l
Create, update and review Support cases
View the Global Support Guide for a detailed explanation of support programs, online services,
contact information, policies and procedures. The guide is available at: www.quest.com/support.
Spotlight Support
When contacting Quest Support about a Spotlight product, please have the version and build
number handy. You can obtain both by choosing the About option from the Help menu in the
relevant Spotlight application. If your question is about an error message, write the message
down and have it available for the technical support representative.
Whenever you have a question about Spotlight, click Support Bundle or Contact Support on
the Help menu. This creates a file called SpotlightSupport.zip. This file contains a snapshot of
your Spotlight installation. Send this file and your request for assistance to Quest Software
(support@quest.com).
Note: You can find SpotlightSupport.zip in the following folder:
l
C:\Users\<user>\AppData\Roaming\Quest Software\Spotlight\Default\Support
l
For Windows XP / Windows Server 2003 the folder is C:\Documents and
Settings\<user>\Application Data\Quest Software\Spotlight\Default\Support
If you find that information in the Help system or printed user guides is incorrect, unclear, or
incomplete, please report the problem to Quest Support.
Spotlight on SQL Server Enterprise Getting Started Guide
32
Appendix: Contact Quest
Contact Quest Software
Email:
info@quest.com
Mail:
Quest Software, Inc.
World Headquarters
5 Polaris Way
Aliso Viejo, CA 92656
USA
Web site:
www.quest.com
Refer to our Web site for regional and international office information.
About Quest Software
Now more than ever, organizations need to work smart and improve efficiency. Quest Software
creates and supports smart systems management products—helping our customers solve everyday
IT challenges easier and faster. Learn more at www.quest.com.
P
Index
A
about Quest Software
29
playback
10
Q
32
B
blocking
performance counters missing
8
Quest Software
32
Quest support
31
R
C
connection limit
28
connections
record and playback
10
rowset trace
13
S
issues
24
troubleshoot
24
server side trace
13
Contact Quest Software
32
session trace errors
29
counters
29
snapshots
10
Spotlight browser
10
CPU usage
5
Customer support
31
D
database growth
18
SQL
expensive
12
long running
21
SQL Analysis
E
error codes, WMI
25
configure
12
expensive SQL
12
filter data
16
SQL Server
H
history
10
connection limit
28
view
10
performance counters
29
history browser
10
Support
T
I
installation issues
24
IO bottleneck
6
L
live view
31
10
Technical support
31
troubleshoot
connections
24
installation
24
Spotlight on SQL Server Enterprise Getting Started Guide
Index
performance counters
29
Playback
30
session trace
29
unable to perform certain operations
28
user groups
28
WMI
25
V
view
history
10
Virtualization Overhead gauge
22
VMware
connection
23
W
wait statistics
20
WMI issues
25
34