WW TSS 10 - Historian 2012 R2 Architecture and

Slide 1
WW TSS-10 - Historian 2012 R2
Architecture and Troubleshooting
Paul McGuire
Jim Stella
social.invensys.com
@InvensysOpsMgmt / #SoftwareRevolution
/InvensysVideos
/Wonderware
/company/Wonderware
© 2013 Invensys. All Rights Reserved. The names, logos, and taglines identifying the products and services of Invensys are proprietary marks of Invensys or its subsidiaries.
All third party trademarks and service marks are the proprietary marks of their respective owners.
Agenda
•
From Historian 10 to Historian 11
•
HCAL
•
Historian Project Tracking
•
Troubleshooting Tools and Tips
Slide 3
The Wonderware Historian Solution
The Wonderware Historian is a high-capacity, high-throughput server
that stores time-series data. The historian acquires and stores process
data at full resolution or at a specified resolution and provides
real-time and historical data together with configuration, event,
summary, and associated production data to client applications on the
desktop. The historian combines the power and flexibility of Microsoft
SQL Server with high speed acquisition and efficient data
compression.
Slide 4
64-bit Microsoft SQL Server Support
Historian Architecture
• OLE DB Provider runs in SQL Server process
• Some interprocess communication via shared memory
• 64-bit SQL Server  64-bit Historian
(mostly, but not all)
Solution in Historian 2012 R2:
• Native 64-bit Microsoft SQL Server Support
• Native 64-bit Historian OLE DB Provider
• 64-bit Process Address Space
• Ability to utilize all installed RAM
Higher scalability will require 64-bit
Phasing out support for case-sensitive collation
Slide 5
Storage Engine Retrieval Performance
Legacy
Storage
>2x
faster
Storage
Engine
Retrieval time for 3-months of
1-second data for 1 tag using “best
fit” with 1-hour period*
*YMMV
Slide 6
Engine Data Acquisition Throughput
150,000
100,000
10.0
2012 R2
50,000
0
Sustained
Burst
Values Per Second
Slide 7
Late
Historian 10.0 Architecture
SQL Server
Retrieval
IDAS/
SuiteLink
History Blocks
Storage
Engine
“Pull” Data
Acquisition
Storage
“Push” Data
Acquisition
Replication
Acquisition
Application
Server
Historian
New “historian”
hidden within 10.0
Slide 8
Historian 2012 R2 Architecture
SQL Server
Retrieval
IDAS/
SuiteLink
Most changes apply only here
History Blocks
“Pull” Data
Acquisition
“Push” Data
Acquisition
Application
Server <3.6
Storage
Engine
Classic
Storage
Application
Server >3.5
“Push” Data
Acquisition
Replication
Acquisition
Historian
Slide 9
Historian 10.0 Storage Data Flow
CSV
Historian 10.0
named pipes
outside RTW
Real-time Storage
Manual Storage
inside RTW
IDAS
MDAS (DCOM)
Configuration
Service
AI
History
Blocks
WAS 3.1 / SDK 1.0
Microsoft TDS
OLE DB Provider
(SQL)
ADO / SSMS
MDAS2 (TCP/IP)
Tier-1
Slide 10
MDAS
Server
Tier-2 Storage Engine
(aahStorageEngine.exe)
S/F
Historian 11.0 Storage Data Flow
CSV
Historian 11.0
named pipes
Classic
Real-time Storage
IDAS
MDAS (DCOM)
WAS 3.1 / SDK 1.0
Configuration
Service
outside RTW
inside RTW
Classic
Manual Storage
AITag=0
AI
OLE DB Provider
(SQL)
AITag=0
ADO / SSMS
HCAL (TCP/IP)
Tier-1
WAS 2012 R2
SDK 2012 R2
Historian Client
Access Point
(HCAP)
Historian Storage
(aahStorage.exe)
AITag=0
Slide 11
AITag=1
History
Blocks
AITag=1
Microsoft TDS
S/F
No Real-time Window Restriction for Data in the Past
• Real-time window check for data in the past is removed. But time in
the future is still overwritten.
• After reconnecting to Historian, HCAL will keep the tag in
store/forward until its value’s timestamp exceeds the “Cut-off” time
to ensure that the store/forward data can cover QD10 (disconnect)
injected on HCAP.
• Cut-off time is the latest of the following:
• Configuration service start time
• HCAP start time
• The tag’s disconnect time (QD10’s time stamp)
• Cut-off time is retrieved for each tag when HCAL reconnect with
HCAP.
• While staying in store/forward, the tag’s value can still be forwarded
to Historian.
Slide 12
Injecting Channel Information
• QD 24 – NULL (HCAL), real-time tag (Channel Status = 1).
Condition - Application Engine is set off scan or disconnected.
Value – Manufactured empty value with current time stamp.
• QD 248 – Store/forward start (HCAL), all except replication tags.
Condition – Storage switches to Store/Forward.
Value – Duplicate of last value sent to server.
• QD 44 – Reconnect (HCAL), all except replication tags.
Condition – Storage switches to server.
Value – Duplicate of last value sent to store/forward.
• Adjustment – Time may be adjusted by HCAP to be 1 msec after timestamp of QD10 if
needed.
• QD 10 – Disconnect (HCAP), real-time tag (Channel Status = 1).
Condition – HCAP lost HCAL heartbeat for more than 20 seconds and values had been
sent to storage since last disconnect.
Value – Last value sent to storage with current time stamp.
Slide 13
MDAS2 in Replication (Historian 10)
Slide 14
HCAL Components
• Historian Client Access Layer (HCAL) consists of the following:
– Historian Client Access Library (HCAL) –
aahClient.dll + aahClientCommon.dll,
static libraries, native, and managed versions
(SDK)
– HCAL Multicast Component (HCAL MCC) – aahClientMCC.dll (Dual Historian Support)
– Historian Client Access Point (HCAP) - aahClientAccessPoint.exe (Historian and
Redundant Platform)
• HCAL/HCAP is based on MDAS2/MDASServer.
• HCAL are used by the following applications
– Application Engine – aahClientMCC.dll + aahClient.dll + aahClientCommon.dll for
Historian access
- aahClientmanaged.dll (SDK) as library for scripting.
– Replication Service (DLL) – aahClient.dll, aahClientCommon.dll (retrieval and storage)
– OLEDB (Static Library) – aahStaticClientNative.lib, aahStaticClientCommonNative.lib
– SDK (Managed DLL) – aahClientManaged.dll
Slide 15
Connection To Historian
Remote Computer
Local Computer
HCAP
other Historian
processes /
subsystems
HCAP
other Historian
processes /
subsystems
Client Application
Named
Pipe
HCAL
Slide 16
TCP
Net TCP Port
Sharing Service
(SMSvcHost.exe)
Connection Management
HCAL
•
Historian connection status is determined by communication layer.
• Disconnected if:
• A function returns network related error or timed out for storage (Typically
1 minute, 10 seconds for sending values)
• Communication layer generates exception
• Ping and report tag count/data rate to Historian periodically (every 3
seconds)
• Similar mechanism is used to manage connections to local Storage Engine
and standby Storage Engine
HCAP
• Client considered disconnected if no pinging from client for more
than 20 seconds.
Slide 17
Store/forward
• Store/forward values will be flushed to disk and forwarded if HCAL is
connected Historian, all tags had been synchronized, and also one of the
following is true:
• No value has been sent to store/forward engine for more than 30 seconds.
• More than 1 hour since last flush.
• Snapshot memory consumption reaches 64 MB.
•
The S/F data gets extracted from the S/F storage engine blocks by HCAL and
sent as S/F snapshots to the historian (first to HCAP and then to Storage)
then the snapshot gets marked as forwarded on the client.
Eventually the whole orignial.dat file on the client side gets deleted when all
its snapshots have been forwarded.
Slide 18
Store/forward and Forwarding
• A lock file “HCALStoreforward.lok” in store/forward directory is used
by HCAL for the following purposes:
• Ensure that the store/forward directory is used by HCAL exclusively.
• Contains base key number for local key generation (redundancy support).
• Passing parameters (e.g. partner Historian) across redundant platform.
• Information for diagnostics. (e.g. computer name, process id etc.)
• Store/forward engine flushes both tags and values to disk.
• Data file (original.dat) is deleted after forwarding completed.
• Tag information with server tag Id will remain and will be loaded
when HCAL restarts.
• Need to delete the store/forward directory for a “Clean” start.
Slide 19
Redundancy Support
• HCAL communicates with its partner store/forward engine through
HCAP.
• One instance of HCAP manages all store/forward engines on
redundant platform.
• Tags are always synchronized between partner store/forward
engines, even when HCAL is connected to Historian.
• Values sent to local store/forward are also sent to partner
store/forward engine.
• Before forwarding values to Historian, HCAL synchronizes values
between the partner store/forward engines.
• After values are forwarded, snapshots on both store/forward engines
are deleted.
Slide 20
Redundancy Support
c o n tr o l v ia T C P / IP
a ah C lien tM C C .dll
c o n tr o l v ia T C P / IP
aahClient.dll +
aahC lientCom m on.dll
a a hC lie nt.dll +
a a hC lie ntC o m m on .d ll
HCAL
A p pica tion S e rver E n g in e P roce ss
B a c k u p P la tfo r m
HCAL
a a hC lie ntM C C .d ll
A p pica tion S e rver E n g in e P roce ss
P r im a r y P la tfo r m
s ta r t / s to p / s to r e /
g e t s n a p s h o ts /
m e ta d a ta
s ta r t / s to p / s to r e /
g e t s n a p s h o ts /
m e ta d a ta
S to r e /fo r w a r d
T 2 E C h ild P r o c e s s
HCAP
s ta r t / s to p / s to r e /
g e t s n a p s h o ts /
m e ta d a ta
S /F d a ta
S /F T 2 E d a ta
Slide 21
HCAP
s ta r t / s to p / s to r e /
g e t s n a p s h o ts /
m e ta d a ta
S to r e / fo r w a r d
T 2 E C h ild P r o c e s s
S /F d a ta
S /F T 2 E d a ta
Historian Project Tracking
•
Provide statistics on general growth and health of the deployment of
the system
•
Useful to step back and appraise overall data collection application
•
Can provide an “executive dashboard” for communication with
customers, peers, management
Slide 22
Statistics for Monitoring/Reporting
Project Deployment
Activity levels
tasks/time metrics
Resource consumption
disk storage
CPU (inferred from activity)
license
Utilization metrics
defined vs. used
Factory coverage
number of sensors/tools
Slide 23
At-a-Glance Dashboard
Slide 24
Collecting the Statistics
The “AAA”-related metrics are gathered from data tables maintained by
the “AAA” application
“DefinedTags” is SELECT COUNT(*) FROM TAG
“ValuesPerSecond” is avg(value) of SysDataAcqOverallItemsPerSec
sampled during the day
“Disk Space Free” is the value of SysSpaceMain
Other metrics
“Tags used” inferred from daily history block’s internal file
“Daily Storage” is the size of daily history block
Slide 25
Project Metrics as a Tool
•
Forces you to think “big picture”
•
Useful format for reporting to customers, operations management,
systems management
•
Format is familiar to management when budgeting for new
hardware, licenses, software modules
•
Useful when planning new deployments or applications
• How will the new project affect daily
activity/consumption/performance?
•
Becomes a common basis for
discussion across operations,
systems, support, and management
Slide 26
Troubleshooting Tools
•
Historian Client
•
SQL Management Studio/Profiler
•
SMC log flags
•
Regedit
•
Process Explorer
•
Performance Monitor
•
PSSDiag and PAL Tool
Slide 27
SMC logger
• Connection Related
• Turn on “Connection” log flags to see WCF messages on client computer.
• Check warning messages on Historian computer.
• Could be caused by firewall blocking, incorrect port, invalid credentials etc.
• Tag Related
• Check to see if there is warning messages in the logger.
• Toggle “TagDump” log flags to see HCAL cached tags.
• Value Related
• Turn on “BufferTransmit” flag to see if all tags are out of store/forward
Slide 28
SMC logger
• S/F - StoreForward log flag in the aahClientCommon section
• Turn on log flags for the right component
• Retrieval with OLEDB – aahClientCommonNative
• Entry-Exit – the hosting process, e.g. SqlServer, AppEngine
• Be aware of the user account the application is running with. HCAL
might fail to write to the tag trace file.
Slide 29
Perfmon
Performance Troubleshooting - Memory
Following counters should be
added to perfmon DataCollection
to identify the memory issue (the
star indicates all counters in the
class)
Memory\Available KBytes
Memory\Cache Bytes
Memory\Commit Limit
Memory\Modified Page List Bytes
Memory\Pool Nonpaged Bytes
Paging File(\??\C:\pagefile.sys)\*
Process(*)\Private Bytes
Process(*)\Working Set
SQLServer:Buffer Manager\Free pages
SQLServer:Buffer Manager\Page life expectancy
SQLServer:Memory Manager\Target Server Memory
(KB)
SQLServer:Memory Manager\Total Server Memory (KB)
Slide 30
PSSDiag/PALtool
Performance Troubleshooting
Slide 31
PSSDiag
• PSSDiag is an application that provides GUI to SQL server
troubleshooting tools. It can be used to define e.g. trace templates or
performance counters logs. This application is used by Microsoft
support and is able to log 32 bit SQL server counters as well.
• For details on using PSSDiag see the link:
http://blogs.msdn.com/b/pamitt/archive/2011/02/25/how-to-use-thesqldiag-the-sqlnexus-and-the-pal-tools-to-analyze-performanceissues-in-sql-server.aspx
Slide 32
PSSDiag
• The tool is started by running Pssdiag.cmd that is found in CAB file
created in the UI.
• By default it is found in C:\Program Files (x86)\Microsoft\Pssdiag\Customer
• There is a little bug in the CMD file so it has to be edited in Notepad because
the path is missing a backslash and also is not customized for 64 bit OS
• The line should look like this:
if "%toolsbin%"=="" set toolsbin=C:\Program Files (x86)\Microsoft SQL
Server\100\Tools\Binn\
Slide 33
PAL (Performance Analysis of Logs) Tool
Slide 34
PAL (Performance Analysis of Logs) Tool
•
Easy to use tool fo analyzing performance counter logs
•
Uses predefined trashold files for different types of analyzes (eg.
SQL Server, System Overview, Exchange Server...)
For download and tutorials:
http://pal.codeplex.com/
Slide 35
Values are not right, or?
Slide 36
Slide 37
Values are not right
Slide 38
Values are not right - tracing
Slide 39
Values are not right - tracing
Slide 40
Tracing Storage
Slide 41
Tracing Storage
Slide 42
Tracing Storage
Slide 43
DEMO
Slide 44
Question/ feedbacks/ request
Thanks !!!
Slide 45
Slide 46