Real-Time High Performance Computing Platform for Smart

Real-Time High Performance Computing Platform
for Smart Grid Applications
Ganesh Kumar Venayagamoorthy, Senior Member, IEEE
Bipul Luitel, Member, IEEE
Real-Time Power and Intelligent Systems Laboratory Real-Time Power and Intelligent Systems Laboratory
Holcombe Dept. of Electrical and Comp Engineering Holcombe Dept. of Electrical and Comp. Engineering
Clemson University
Clemson University
Clemson, SC 29634 USA
Clemson, SC 29634 USA
gkumar@ieee.org
iambipul@ieee.org
Abstract—Real-time control of smart grid will require fast and
accurate implementation of intelligent monitoring and decision
making techniques. Intelligent methods with faster speed and
better accuracy will be possible by implementing in parallel on
high performance computing platforms. Such methods need to
be rigorously tested in simulations before implementing in actual
systems. Therefore, integration of simulation environments with
high performance computing (HPC) platforms is necessary to
develop and test real-time monitoring and control methods for
smart grids. Such integration is a challenge because of hardware
and software limitations of real-time digital simulator (RTDS),
HPC platforms as well as physical inaccessibility of HPC systems.
In this paper, an integrated platform consisting of RTDS, HPC
and data acquisition system is developed for real-time streaming
of simulation data for computation. A wide area monitoring
system for a multi-machine power system is illustrated on such
integrated platform.
Index Terms—HPC, Real-time simulation, RTDS, Smart grid,
WAMS
I. I NTRODUCTION
Addition of renewable resources, distributed generation and
bidirectional power flows in the smart grid will make it more
complex than ever. Smart grid will have high uncertainty and
variability owing to high penetration of intermittent energy
sources. Therefore, it is necessary to have real-time simulation of power system phenomena, and related systems of
communication and control under these changed scenarios in
order to ensure stability, reliability, integrity and security of
the electric power grid. Real-time simulation is a computer
model of a physical system that can be executed at the same
rate as the actual system. Real-time simulator is a combination
of computer hardware and software that enables real-time
simulation. Because of the deterministic times for simulation,
real-time simulators are used in various engineering fields,
and more so in the simulation of mission and time critical
applications. Real-time digital simulators (RTDS) are a set of
hardware platforms and accompanying software suite for simulation of power systems [1]. These allow for the simulation
of power system and control components to evaluate system
performance under various operating conditions, and to test
devices and control systems. Real-time simulation of power
system has been of interest for quite some time now [2]. Today,
RTDS are widely used by industries and universities across
the world [3], [4]. Power system equipment manufacturers,
electrical utilities and research institutes benefit from the realtime simulation technology which allows them to test and
implement new ideas, including closed loop testing of physical
hardware, before integrating with the actual system [5]. Many
researchers over the past decade have used RTDS as a power
system test bed for real-time power system simulation studies
[6], [7].
R
is a real-time hardware platform used in power
RTDS
system simulation. It is widely used for simulation of various
power system components, phenomena and control. As smart
R
is
grid research has taken a leap in the past few years, RTDS
also being used in simulation of distributed power grids [8],
for the study of HVDC grid [9], and several other smart
grid applications such as simulation studies on integration
and testing of phasor measurement units [10]–[12]. Thus,
R
RTDS
has proven to be an important tool for successful
deployment of smart grid technologies. Real-time digital simulators also enable hardware-in-the-loop based simulations.
These are useful for implementing parts of the physical system
in simulation and still operate as a complete physical system.
Such advanced features allow fast integration of new technologies into the smart grid. Real-time simulation also allows
testing of various operating conditions and phenomena that
cannot be realistically implemented on the physical system.
But as the complexity of physical system increases, so does
the complexity of monitoring and control methods. In the
future, intelligent monitoring and control of smart grid will
be carried out on high performance computing (HPC) platforms using computational intelligence methods implemented
using parallel programming [13]. Therefore, it is important
to have a closed loop communication between the HPC and
the RTDS for simulation and study of intelligent monitoring
and control techniques implemented on HPC to be used in
smart grid applications implemented on RTDS. Until now,
such implementation has been challenged by the following
problems:
1) RTDS and HPC infrastructures are not physically collocated (and may be several miles apart) to share appropriate signals between them.
2) HPC cluster consists of several computing nodes and
which nodes will be used for a particular computation
is non-deterministic.
3) RTDS does not natively support sharing raw data over
the Ethernet for the purpose of wide area monitoring
and control.
In this paper, a method for using HPC in closed loop
R
communication with the RTDS
is presented. The proposed
real-time high performance computing (RT-HPC) platform is
used to implement a wide area monitoring system (WAMS) for
smart grids. The remaining sections of the paper are arranged
as follows: different components of the RT-HPC platform
are described in Section II. The wide area predictive state
estimation is described in Section III. The results of the study
are shown in Section IV and conclusions in Section V.
II. T HE RT-HPC P LATFORM
In an actual physical system, the health of the system is
assessed by monitoring different parameters of the system.
Based on the values of these parameters, decisions are made
and control signals dispatched to drive the system from one
operating condition to the other or keeping it stable. Therefore,
decision making and control systems need to know the status
of the system. In a power system, certain events take place very
fast and therefore fast decision making and control systems are
required for real-time control. With increase in system complexity, more complex decision making and control systems
will be required. These systems need to be implemented in
parallel using high performance computing platforms in order
to meet the timing requirements of real-time monitoring and
control. Therefore, in order to simulate and study intelligent
real-time monitoring and control techniques utilizing HPC, a
platform for integrating simulation environment with computing resources is required. The RT-HPC platform provides the
R
ability to share data between the RTDS
and the HPC.
Power systems are modeled on RSCAD and simulated on
R
RTDS
. Intelligent monitoring and control algorithms that
are implemented on HPC require the data generated from the
simulation in real-time. However, HPC clusters are generally
physically inaccessible. HPC clusters may consist of thousands
of individual nodes connected to each other and it is not known
which node is going to be utilized when a program is executed.
Therefore, it is impractical to share data between the RTDS
and the HPC via direct physical connection. Therefore, in
the platform developed in this study, data is shared between
the two via TCP sockets using the Labview programming
environment and PXI controller hardware as the intermediate
R
component between the RTDS
and the HPC cluster. The
different signals of interest in the simulation are connected
R
between the RTDS
and the PXI controller using physical
wires. A program written in Labview software on the PXI
controller then communicates the signals over the Ethernet.
The output of monitoring and control algorithms from the HPC
R
cluster is then transmitted back to the RTDS
using the reverse
path. The RT-HPC platform with all the components is shown
in Fig. 1.
A. Real-Time Digital Simulator
Real-time digital simulator used in this study is the
R
RTDS
from RTDS Technologies. A single rack of the RTDS
is shown in Fig. 2. The RSCAD graphical user interface
(GUI) module provides design tools and component libraries
for creating a model of the power system to be simulated
R
on RTDS
. Fig. 3 shows the Draft case of the test system
modeled on RSCAD. The draft is then compiled and loaded
R
at 50 µS time step (i.e. 20 KHz).
and executed on the RTDS
The Runtime GUI module provides visualization and control
interface for the simulation running on the RTDS. This module
consists of GUI tools for the components included in the Draft
case. Fig. 4 shows the Runtime window with the controls and
graphs for the simulation.
R
Each rack of the RTDS
consists of processing, communication and interface cards. In this study, two racks consisting
of three Gigaprocessor cards each are used for running the
test system. The racks also consist of analog and digital
input/output (I/O) cards. RSCAD library provides with components that can be used to receive/send analog and digital
signals to/from the simulation. The analog input cards of the
RTDS , called GTAI, are used to receive signals into the
simulation, and the analog output cards, called GTAO, are used
to to send them from the simulation. This is done by analog
to digital (A/D) and digital to analog (D/A) converters on the
I/O cards. The GTAI/O cards provide analog interface to the
external devices for hardware-in-the-loop studies. Fig. 5 shows
the GTAI and GTAO cards consisting of 12 channels each. In
R
this study, voltages of 12 buses is sent from the RTDS
to
the HPC cluster. Therefore, 12 analog channels from a GTAO
cards are used. Similarly, the predicted bus voltages are sent
R
back to the RTDS
from the HPC cluster for visualization
purpose, hence using 12 channels of two GTAI cards.
B. PXI Controller
The PXI controller from National Instruments (NI) is a
computer with an i5-2510E 2.5 GHz processor running Windows 7 32-bit operating system. The PXI controller hosts
NI’s Labview development suite. Apart from the controller,
the PXI chassis consists of several slots that can be used to
add analog and digital I/O cards. In Fig. 6, the configuration
of PXI controller used in this study is shown. It consists of
two analog input (AI) and two analog output (AO) cards. Each
card consists of 32 analog channels for external interface. The
18 channels of the AI card on the PXI are connected to the 18
R
channels of the two GTAO cards of the RTDS
by physical
wires. Similarly, the 15 channels of the AO card on the PXI
are connected to the 15 channels of the two GTAI cards. A
Labview program is written to read/write the signals from/to
the I/O cards connected to the PXI controller, using which
a two-way communication between the RTDS and the PXI
controller is established.
Another task of the PXI controller is to communicate the
R
signals received from the RTDS
to the HPC cluster, and
vice versa. Since both the cluster and the PXI controller are
connected to the Ethernet, the communication is performed
Fig. 1.
Real-time high performance computing platform showing the different components of the platform.
R
rack showing processing, communication and
Fig. 2.
A single RTDS
interface cards.
Fig. 4. The Runtime module of the RSCAD showing control and visualization
components.
Fig. 5.
Fig. 3. The Draft module of the RSCAD window showing the power system
model and component library.
R
.
The GTAI and GTAO cards of the RTDS
in real-time using TCP sockets. The Labview program implements a TCP server and listens to a specific communication
port. Data is then shared between the server and any client
that connects to the port. The TCP read/write program works
like a file read/write to a communication port, and hence the
shared data is in string format. Therefore, the data from the
Fig. 6. The PXI chassis showing the controller, and analog input and output
cards.
!
"
#$ $%%
between the processors, is used to implement the WAMS in
this study. The program runs in parallel on multiple processors
and data among the processors is shared using send and
receive functions available in MPI libraries. The program
running on the master processor acts a a TCP client and
connects to a specific port on the TCP server, which is the
PXI controller. Thus, the master processor is able to establish
a two-way communication with the PXI controller. Once the
concatenated string of data is received, it is shared with the
other processors using MPI after splitting and converting it to
a numerical value. The master processor also collects the data
from different processors, converts it to string and concatenates
before sending it the PXI controller using TCP. Thus, the
algorithm for predicting the voltages at different buses running
concurrently on different processors is able to utilize the data
from the RTDS received via PXI controller.
III. W IDE A REA P REDICTIVE S TATE E STIMATION
(* +%,
(#)
' % $-
' % (#)
(* +%,
$-
& % #$
$
Fig. 7.
The block diagram shows different operations performed using
Labview on the PXI controller.
different signals is first converted to string and concatenated
before sending it via TCP ports. Similarly, a concatenated
string received from the cluster is first split and converted to
numerical value before sending it back to the RTDS via AO
cards. The Labview program is shown in Fig. 7.
C. High Performance Computing Platform
The high performance computing platform used in this
study is the Clemson University Palmetto cluster located at
Pendleton, SC approximately 10 miles from where the RTDS
and the PXI controller are located. It consists of 1,681 compute
nodes with a total of 15,120 cores, and benchmarked operation
at 96 TF/S. The cluster operates on Scientific Linux operating
system and uses PBS job queuing system [14]. The different
compute nodes use Infiniband and Myrianet network fabrics
to communicate with each other.
Each user program that needs to run on the cluster is
considered as a job and is handled by the PBS. The number
of processing units requested by the user in each job may be
allocated from any combination of available compute nodes
in the pool. A program written in C using message passing
interface (MPI), a set of libraries that enable communication
The test system used is a 12-bus system shown in Fig. 8
consisting of three generators [15]. Prediction of power system
variables allows a wide area monitoring system to estimate
the system states ahead of time. In this study, a WAMS for
R
the test system simulated on RTDS
is implemented on the
HPC platform. The WAMS is developed using computational
elements arranged as cells that connect to each other in a
certain fashion. The WAMS consists of 12 cells, each cell
representing one bus of the system. The computational units
predict the voltage of the bus they represent based on the past
values, and predicted time-delayed inputs from the connected
buses. The computational units of the WAMS are implemented
using neural networks in this study [16]. The WAMS is
thus developed based on the connectivity matrix of the buses
and the generators of a power system. A WAMS developed
for bus voltage prediction of 12-bus system is shown in
Fig. 9. The figure shows that each bus is represented by a
cell which is connected to the other cells representing the
connected buses. Thus it is able to capture the topology of
the system. The WAMS is implemented on HPC where each
cell is implemented on a different processor and executed in
R
parallel. The output of the WAMS is sent to the RTDS
and
visualized on the Runtime. The predicted output of WAMS can
be further utilized for situational awareness, decision making
and predictive control in smart grids.
IV. R ESULTS
In this study, the test system is perturbed by applying small
disturbances to the excitation system of the generators. Since
the operation takes place in real-time, the prediction algorithm
utilizes past values of the arriving signals and predict the
current values (as opposed to using current values to predict
the future values) for visualization and comparison with the
actual values. In order to see the ability of the WAMS to
predict the behavior of the system during faults, a 10-cycle
three phase short circuit is applied to bus five of the system,
and the response of the WAMS is captured along with the
response of the system. Figs. 10 and 11 show the comparison
of actual against the predicted bus voltages when a small
disturbance is applied and during a three-phase short circuit
fault, respectively. These results are captured in the Runtime of
RSCAD when the WAMS is running on the Palmetto cluster.
Since the captured signal has a time step of 10 ms, the total
time for computation and communication needs to be less than
10 ms in order to visualize without a delay. The platform
developed in this study is able to satisfy this requirement.
tŝĚĞƌĞĂDŽŶŝƚŽƌŝŶŐ^LJƐƚĞŵ
'ϭ
ϵ
V. C ONCLUSIONS
ϱ
Ϯ
ϰ
ϭϬ
'Ϯ
ϭ
ƌĞĂϯ
ϭϮ
ϲ
ƌĞĂϭ
'ϰ
ƌĞĂϮ
ϴ
ϳ
ϯ
ϭϭ
'ϯ
Fig. 8.
IEEE standard 12-bus system used in the simulation.
R
An integrated platform consisting of RTDS
, the HPC
cluster, and a data acquisition system is developed for streaming real-time simulation data for computation is developed.
Real-time implementation of intelligent monitoring and control techniques in high performance computing platform is
demonstrated. The data required by the algorithm running
on HPC cluster is provided in real-time through the power
R
via Ethernet using Labview
system being simulated on RTDS
implemented on a PXI controller. As the size and complexity
of modern power system increases, use of real-time high
performance computing platforms becomes more significant.
In the future, detailed study on timing based on distance
between the simulation and computation platforms need to be
performed in order to identify the limitations due to communication. The study presented in the paper also opens future
area of research on applications of real-time high performance
computing platforms in smart grids.
ACKNOWLEDGMENT
The funding provided by National Science Foundation, USA
under the CAREER grant ECCS #1231820, #1232070 and
EFRI #1238097 is gratefully acknowledged.
tD^
92
2
R EFERENCES
./
98
93
3
9/
/
9/4
.3
91
8
1
/4
95
9/3
/3
5
ƌĞĂϭ
97
7
.1
ƌĞĂϯ
96
90
ƌĞĂϮ
.0
9//
//
6
0
Fig. 9. A wide area predictive state estimator overlaid on top of the 12-bus
system for bus voltage prediction.
[1] R. Kuffel, J. Giesbrecht, T. Maguire, R. Wierckx, and P. McLaren,
“RTDS - a fully digital power system simulator operating in real time,”
in IEEE Communications, Power, and Computing Conference, vol. 2,
1995, pp. 300–305.
[2] M. Rafian, M. Sterling, and M. Irving, “Real-time power system simulation,” IEE Proceedings of Generation, Transmission and Distribution,
vol. 134, no. 3, pp. 206 –223, May 1987.
[3] RTDS, “Clients of RTDS technologies,” http://www.rtds.com/
clients-rep/clients.html,Last accessed: Nov 21, 2012.
[4] OPAL-RT, “Clients of Opal-RT technologies,” http://www.opal-rt.com/
Customers,Last accessed: Nov 21, 2012.
[5] P. Forsyth, T. Maguire, and R. Kuffel, “Real time digital simulation
for control and protection system testing,” in IEEE Power Electronics
Specialists Conference, vol. 1, June 2004, pp. 329 – 335 Vol.1.
[6] I.-Y. Chung, W. Liu, K. Schoder, and D. A. Cartes, “Integration of a bidirectional DC-DC converter model into a real-time system simulation
of a shipboard medium voltage DC system,” Electric Power Systems
Research, vol. 81, no. 4, pp. 1051 – 1059, 2011.
[7] P. Tatcho, Y. Zhou, H. Li, and L. Liu, “A real time digital test bed
for a smart grid using RTDS,” in Symposium on Power Electronics for
Distributed Generation Systems, June 2010, pp. 658 –661.
[8] K. Ravikumar, N. Schulz, and A. Srivastava, “Distributed simulation of
power systems using real-time digital simulator,” in IEEE/PES Power
Systems Conference and Exposition, March 2009, pp. 1 –6.
[9] P. Mitra, K. Vinothkumar, and L. Zhang, “Dynamic performance study
of a HVDC grid using real-time digital simulator,” in Complexity in
Engineering (COMPENG), 2012, June 2012, pp. 1 –6.
1.02
1.0396
0.985
0.98
0.99
0.98
V4
V3
V2
V1
1
1.0394
0.98
1.0392
0.965
0.97
1.04
0.98
1.04
0.97
V8
V7
V6
V5
1.0402
0.99
1.03
1.02
0.99
0.96
0.97
1.01
1.04
1.01
1
1
V 12
V 11
1.02
V 10
V9
0.98
1.0398
1.01
1.04
1.04
1.04
1.04
1.04
0.975
0.97
1
0.99
0.98
:;<=>?
0.99
0.98
0
@ABCD;<BC
5
9
Fig. 10. Bus voltage prediction from WAMS compared against the actual signals when a small disturbance is applied to the excitation system of the
generators.
0.98
V6
V5
V3
0.9
1.037
1
0.8
0.6
0.4
0.2
0.95
V7
1.038
V2
V1
1.039
0.96
0.94
0.98
0.96
0.94
0.92
0.9
0.88
1.041
1.04
1.039
1.038
1.037
V4
1
0.95
0.9
0.85
0.8
V8
1.04
0.98
0.96
0.94
0.92
0.9
0.88
1.0401
0.95
1.04
1
0.98
0.96
0.94
0.92
1
V12
V10
V9
1.04
V11
1
1.04
0.9
0.96
EFGHIJ
Fig. 11.
0.98
KLMNOFGMN
0
5
9
Bus voltage prediction from WAMS compared against the actual signals when a 10-cycles three phase short circuit is applied at bus five.
[10] C. Liu, Z. H. Rather, N. Stearn, Z. Chen, C. L. Bak, and P. Thogersen,
“Practical testing and performance analysis of phasor measurement unit
using real time digital simulator (RTDS),” in IEEE International Energy
Conference and Exhibition (ENERGYCON), Sept. 2012, pp. 408 –414.
[11] H. Chen, B. Bhargava, F. Habibi-Ashrafi, J. Park, and J. Castaneda,
“Integration of RTDS with EPG synchrophasor applications for visualization and analysis of simulation scenarios at Southern California
Edison,” in North American Power Symposium, Sept. 2012, pp. 1 –5.
[12] D. S. Ouellette, M. D. Desjardine, and R. Kuffel, “Using a real time
digital simulator with phasor measurement unit technology,” in 11th
International Conference onDevelopments in Power Systems Protection,
April 2012, pp. 1 –6.
[13] B. Luitel and G. Venayagamoorthy, “Parallelization of cellular neural
networks for smart grid applications,” in Clemson University Power
Systems Conference (PSC12), Clemson, SC, Mar. 2012.
[14] CU CITI, “The palmetto cluster,” http://citi.clemson.edu/palmetto/, last
Accessed: Jan 4, 2012.
[15] S. Jiang, U. Annakkage, and A. Gole, “A platform for validation of
FACTS models,” IEEE Transactions on Power Delivery, vol. 21, no. 1,
pp. 484 – 491, Jan. 2006.
[16] B. Luitel and G. Venayagamoorthy, “Decentralized asynchronous learning in cellular neural networks,” IEEE Transactions on Neural Networks
and Learning Systems, vol. 23, no. 11, pp. 1755 –1766, Nov. 2012.