Lightning Detection System for Ionospheric Studies

Ionosondes Group
Space Science and Technology Department
CCLRC Rutherford Appleton Laboratory
Chilton
Didcot
Oxfordshire
OX11 0QX
http://www.cclrc.ac.uk
http://www.wdc.rl.ac.uk
Lightning Detection
System for Ionospheric
Studies
Nuffield Science Bursary Project
2004
Daniel Wilkins
Supervisor: Dr Chris Davis
CCLRC Rutherford Appleton Laboratory
Table of Contents
1. Introduction............................................................................................3
i.
The Ionosphere.......................................................................3
ii.
The Research Project.............................................................3
iii.
Project Brief.............................................................................4
2. Lightning Detection Systems................................................................5
i.
Hardware Detectors................................................................5
ii.
Software ..................................................................................7
3. Software Design....................................................................................9
i.
Task of Software .....................................................................9
ii.
Main Routines .........................................................................9
iii.
Initial Design of Software......................................................11
iv.
Problems Encountered with Software .................................11
v.
Final Build..............................................................................12
4. Using the Data.....................................................................................14
i.
Using the Data for Research................................................14
ii.
Graphical Display of Data.....................................................15
5. Calibration............................................................................................17
6. Conclusions/Future of the Project ......................................................19
7. Appendices..........................................................................................20
i.
Flow of Data through System...............................................21
ii.
Software Data Acquisition System Flowchart .....................22
iii.
C++ Code for Data Acquisition Tool ....................................23
iv.
Acknowledgements ..............................................................31
2
1
Chapter
Introduction
The Ionosphere
The ionosphere is a layer of ionised gas (produced by many things including the
interaction of radiation, particularly in the ultraviolet part of the spectrum, from the sun
with neutral atoms and molecules), between altitudes of approximately 80 km and 640
km. The ionisation is being continually generated by sunlight and destroyed through
collisions with other ions. The lifetime of individual ions ranges from a fraction of a
second to many hundreds of seconds depending on the atmospheric pressure at each
height.
The ionosphere affects the propagation of radio waves. As the waves interact with the
ions and free electrons, they are partially refracted and reflected back to Earth when
the frequency of the wave is equal to the resonant frequency of the gas; this
phenomenon can have a great effect on radio communications as they alter the range
and clarity of transmitted signals.
This phenomenon can be used to make measurements of the ionosphere using an
ionosonde which reflects radio waves off of the ionosphere and measures what is
received on the ground and from these measurements can determine the height
profile of atmospheric ionisation. The frequency of the radio waves reflected can be
related to the electron concentration at that altitude using the formula:
f = 8.98 [e − ]
Where f is the frequency of the radio wave reflected (in MHz) and [e-] is the electron
concentration (in electrons per m3). This makes it possible to build up a picture of
ionisation against height.
This ionosphere is generally divided into four distinct layers; D, E, F1 and F2
depending on the frequency of radio wave that is reflected or can penetrate the layers.
The lowest frequencies are absorbed by the D layer (which does not always exist) as
electrons take the wave energy and start to resonate but before the wave can be
reradiated, the energy is dissipated by collisions with other gas particles due the higher
pressure at the lower altitude of the D layer. Higher frequency waves can penetrate to
the F layers before being reflected back to Earth. There is also the Es layer (sporadic
E) which is a very dense, patchy layer of metal ions which sometimes appears at an
altitude of around 100 km and reflects a lot of radio waves extending the range
considerably of many transmitted signals.
The Research Project
A lot is still unknown about the factors that affect the composition of the ionosphere
and how energy is transferred between the upper and lower parts of the atmosphere.
This information would be useful as it would allow better prediction of the state of the
ionosphere and to find how, if it is at all possible, energy is transferred between the
upper and lower parts of the atmosphere.
3
This Nuffield Science Bursary project is to be the first step in a research project to find
out exactly what affects the ionosphere, in particular the affect of electrical activity in
the form of lightning strikes on the ionosphere, with particular focus on how electrical
activity affects the composition of the ionosphere.
Project Brief
The Ionosondes group at CCLRC Rutherford Appleton Laboratory have been
collecting Ionospheric data using the digital ionosonde on the Chilton site (and others,
including the one at Port Stanley in the Falklands).
In order to carry out this research, the group require a system to obtain data about
lightning strikes in the vicinity of the Chilton ionosonde which must be able to obtain
data about lightning strikes which can differentiate between strikes regarding the
properties that could potentially cause changes in the ionosphere, including:
•
How far away the strike was
•
The type of lightning strike (cloud-to-ground, intra-cloud or sprite – a strike
that travels up from the cloud to the ionosphere)
•
The polarity of the strike (the direction of electron flow)
•
The potential difference between the cloud and ground, or the ionosphere in
the case of sprites (proportional to the intensity of the strike).
The data about the lightning strikes is to be placed in the World Data Centre database
for analysis.
4
2
Chapter
Lightning Detection Systems
Hardware Lightning Detection Systems
A number of lightning detection systems are currently available on the market. Many of
these work on the principle that lightning strikes generate VHF radio noise which can
be detected in antenna coils within the detector. The direction of the strike is
determined by two crossed coils within the detector and the distance is determined by
the signal strength (which means they cannot be used to determine the intensity of
strikes) as the signal is attenuated as it passes through the air.
Two such systems are available from Boltek (http://www.boltek.com):
Boltek® StormTracker™
This system consists of an antenna unit (black box)
connected to a PCI card and is supplied with simple
software to locate lightning strikes.
The antenna unit can be installed separately to the PC
used to get the data from the device as a long cable is
supplied and extension cabling is available.
• Figure 2.1: Boltek StormTracker™
Boltek® LD-250
• Figure 2.2: Boltek LD-250
This device is similar to the StormTracker™ system but instead of a PCI card has a
separate box which connects to the PC via a USB or RS232 (serial) port.
5
Chosen System
It was decided that the StormTracker™ system should be employed for this project as
it can fit tidily inside a computer and the card itself does not process the data, this is
left to the software (the LD-250 box analyses the data to an extent and may remove
some of the information required about the strikes).
Installation
To ensure accurate detection of strikes and determination of distances and directions
of strikes, the antenna unit is to be installed away from all metal objects (as these can
attenuate the signal making strikes appear further away than they actually are) and
sources of radio noise such as computer equipment and electrical cables (these can
cause non-existent strikes to be detected). To get the best range from the detector, the
antenna unit should be installed as high as possible with a direct line of site to the
horizon.
To achieve the best results, the lightning detector system was installed on the roof of
building R25 with the antenna enclosed in a plastic pipe to protect it, attached to the
side of a fibreglass hut which was to house the computer system to obtain the data
from the detector.
The lightning detector system was installed by Ionosondes group engineer
John Smith and RAL Ground Station engineer John Wright.
• Figure 2.3: Boltek® StomTracker™ (antenna in grey pipe) installed on the roof of building R25.
6
Software
With the Boltek StormTracker™ lightning detector, there are a number of software
options available to obtain the lightning strike data and put it into a form suitable for
this research project.
Boltek StormTracker™
This is the simple package that is supplied with the
StormTracker™ lightning detector that allows only
for simple plotting of strikes on a map and for
tracking thunderstorms. This will not be suitable for
the project as detailed quantitative data about each
strike (including the type and polarity) is not
available.
• Figure 2.4: Boltek StormTracker™ Software
Lightning/2000
• Figure 2.5: Lightning/2000 Software
Lightning/2000 from Aninoquisi (http://www.aninoquisi.com/lightning2000.htm) offers
considerably more than the StormTracker™ software in terms of the data that is
available from it. It works with the Boltek StormTracker™ to provide detailed
information about lightning strikes including the type (cloud-to-ground or intra-cloud)
and polarity of the strike as well as storm tracking facilities and the ability to export
7
screenshots and storm tracking reports, however the raw lightning strike data cannot
be exported from the program.
NexStorm
• Figure 2.5: NexStorm Software
NexStorm from Astrogenic systems (http://www.astrogenic.com/nexstorm) can provide
a similar level of detail about individual lightning strikes picked up by a Boltek
StormTracker™ or LD-250 detector but as well as exporting screenshots and storm
tracking reports (which it generates using its TRAC system – Thunderstorm Ranging
and Acquisition). It also puts the strike data into a shared memory area on the system
so that it can be read by other programs.
It puts all data into the memory but marks those it classes as radio noise not caused
by a strike by setting the distance and bearing of the strike to -1.
Because it makes raw lightning strike data available to other programs via
a shared memory area, NexStorm was selected to get the data from the
detector and a small application will be written to obtain the data and put it
in the World Data Centre database.
8
Software Design
3
Chapter
Task of the Software
In order to utilise the data from NexStorm™ and the Boltek® StormTracker™, an
application was written to take the data from NexStorm’s shared memory space and
output it to the World Data Centre database (using the PostgreSQL SQL database
system) for use in the research project.
The software must:
•
Read the data from memory and output it to the PostgreSQL server without
causing memory access violations or having a too greater toll on network
bandwidth or system resources.
•
Run stably and continuously without any user interaction.
•
Be configurable by the end user (output settings etc.).
•
Be easy to configure and use.
The software was written in the language C++ as this provides a lot of advanced
functions such as memory access and database access (via function libraries for each
database system) as well as native support for Microsoft® Windows® based GUI
(graphical user interface) construction with little complex coding for such trivial tasks
using the Microsoft® Foundation Class library (MFC).
Main Routines
Before work began on the main application to obtain data, research was done into the
options for accessing the memory and the PostgreSQL database.
Memory Access
NexStorm uses a system known as FlashGate IPC (inter-process communication) to
share the lightning strike data using the system memory where the memory location is
accessed by a name so can be in any physical location and still accessed.
Astrogenic Systems provide an example application to access the shared memory
location with C++ source code with NexStorm, which was studied to learn how to
access shared memory under Microsoft® Windows® using the FlashGate IPC
system.
9
• Figure 3.1: NexStorm example FlashGate IPC client program
To access the shared memory area, initially the file mapping object for the memory
area (already created with a specific name by NexStorm) is opened, then the ‘file’ (the
contents of the memory) is mapped into a variable in the address space of the client
program so that the data can be read. The data can then be accessed like any other
variable in that application.
To ensure memory access violations do not occur (which can cause the software to
‘crash’), semaphores can be employed in the application. These are used to control
access to memory locations and to ensure that only one process attempts to access
the data at once. A semaphore is created for the memory location and then the thread
waits for the semaphore to change state signalling that the memory location is ready
for access. When the program has finished reading the memory, the semaphore is
released so that another process can gain access.
All of the data is stored in memory as a single string, which is divided up and put into
the appropriate variables, one for each parameter, by a C function.
A simple console-based application was made to test this code, which reads the
shared memory space and outputs the contents to the console window. This code was
then known to be suitable to be made a module in the final application.
• Figure 3.2: Test console-based application to obtain data from shared memory area.
PostgreSQL Database Access
PostgreSQL provide the well documented libpq function library which contains C
functions for accessing a UNIX-based PostgreSQL database. It is possible to use
10
these function in Windows® based software if the supplied libpq.dll file is
compiled and registered on the machine that will run the software.
To access the database, first a connection is opened to the database on the SQL
server (authenticated with a username and password), and then the SQL command
(INSERT to add data to the database) is passed as a string to a function that executes
the command on the SQL server. The connection to the SQL server is then closed.
The result of executing the function can be obtained to either acquire data or check it
worked.
A console-based application was written to test this code by inputting arbitrary data
into the database. This code was also suitable to be made into a module for the final
application.
Initial Design for the Software
The program will take an object-oriented design with an ‘object’ for the lightning strike
data that can itself obtain the data and output it to the SQL server as well as to a text
file (this was added as a backup for when a network connection to the SQL server is
not available).
The user initially configures the software (how frequently the memory is read, the text
file to put data into and the SQL server connection settings) in a dialog and then opts
to begin data acquisition. From this point, as frequently as set by the user, the software
calls a method of the strike data object that reads the memory and gets the lightning
strike data. It then checks that it is not radio noise not caused by a strike (determined
by NexStorm) or a ‘heartbeat’ from NexStorm (a signal to show NexStorm is still
working) and if it is a new strike (NexStorm keeps the previous piece of data in
memory until more is received). The data is then written, by other methods, to the text
output file and to the SQL server (unless the user opts not to output data to either of
these). This continues until the user sends the command from the dialog to stop data
acquisition.
Problems Encountered with the Software
In the initial build, a number of issues were encountered with the software.
Time Delay Too Long Between Memory Reads
Initially, the software would write to the SQL buffer as soon as it found a strike in
memory. This process can take a considerable amount of time (up to a quarter of a
second for SQL writes) which significantly reduces the resolution of the system
(causing it to miss some strikes).
To overcome this, a technique known as multithreading was employed, where instead
of executing commands one by one in order, the system sends multiple commands (or
‘threads’) through the processor simultaneously. Instead of waiting for one data
acquisition run to complete before starting a timer to begin the next, every so often
(time set by user), a new thread is started to read the memory and to place the data in
the appropriate location(s); a new run can begin before the last had ended.
This approach introduced a number of problems…
11
Data Not Being Placed in SQL Table
Using the multithreading technique, too many commands were being sent too quickly
to the SQL server and it was rejecting some of them, therefore not all of the data was
being recorded in the database.
This was overcome by buffering SQL commands. Where the original build would
concatenate a command then send it to an SQL server, the command would just be
produced then stored in an STL vector (an array of variables stored together as one
unit) of strings (constructs that manage their own memory usage to control use of
system resources). Then, after so many memory reads, when this buffer became full
or when the STOP button is clicked, all of these commands would be sent, as a single
transaction, to the SQL server.
Buffering the commands eliminates the need to start each data acquisition run in a
separate thread (as it is only really the writes to the SQL server that slow the process
down). Removing these threads will increase the reliability of the software as fewer
threads attempting to access the memory will decrease the likelihood of access
violations. The commands will be copied to another, working buffer and sent to the
SQL server in a separate thread so data acquisition can continue while the write takes
place.
High Memory Usage
When monitored with the Windows® Task Manager, the memory usage of the
application appeared to increase rapidly until it stopped at around 130 MB, which is
sufficiently high to imply a problem exists within the code. It also appeared that after a
while, data acquisition would cease, which would suggest that Windows® is not
allowing the software to use any more memory so it cannot continue to operate.
Various parts of the code were studied to find the cause of this problem and it was
found to be caused by opening the mapping object for the shared memory. The file
mapping object would be created each time the memory was read but not destroyed
at the end so several mapping objects would exist filling the memory. To overcome
this issue, the code was altered so to only map the view of the memory space when
the START button is clicked rather than each time the memory needs to be read.
Final Build
After these alterations were made, the user interface of the software was designed to
be as clear and intuitive as possible with the following features:
•
Single dialog box to set up all data acquisition options from one screen using
clear controls before sending the command to begin data acquisition.
•
Tolerance to user error – all options set by user are checked to ensure useful
results will be produced and no system errors will occur (e.g. a valid filename
must be entered and the options for connecting to the SQL server are
checked when data acquisition begins so no time is wasted or data is lost).
•
Because the software is designed to run in the background, the windows was
set to minimise to an icon in the Windows® system tray which changes
depending on what the software is doing. This keeps the tool out of the way
so the user can use other parts of the system (for example NexStorm itself).
12
• Figure 3.3: The main dialog for the Lightning Data Acquisition Tool running on Microsoft® Windows® XP.
An option to log events with a date and time was added to the program to assist with
the debugging procedure to show when the user changes any major settings and
when failures occur.
The flow chart in Appendix ii outlines the workings of the software. The
code of the data acquisition module can be found in Appendix iii.
Testing
Before the final version of this software was compiled and commissioned for use with
the lightning detector it was extensively tested using a range of options including those
that are known to cause errors (such as invalid filenames or SQL server host/port
names that do not exist) and the response of the program checked.
A similar project to detect lightning strikes is being run in Japan
(http://hp.Vector.co.jp/authors/VA034934/Lightning) and those running it have made a
piece of software available to read NexStorm’s
shared memory area and write the data to a
comma-separated text file and to the screen
(NSList) like the tool that has been made. The
data collected from this software was compared
with the data collected by the tool in development
to check how accurate the data was and how
good the resolution was (the number of strikes it
received). Most of the time, every strike picked up
by NSList was written to the database by the tool
in development if the time interval between reads
is set to a sufficiently low value.
• Figure 3.4: NSList for Japanese project
13
Using the Data
4
Chapter
Using the Data for Research
The purpose of the research project is to compare Ionospheric data with that about
lightning strikes in the vicinity of the Chilton ionosonde. To obtain the most conclusive
results, the quantitative data containing the time, position, type and polarity of the
strike will be required; therefore the raw data must be in a usable form in the
PostgreSQL database that the Lightning Data Acquisition Tool placed it into.
Table 4.1 shows the data stored about each individual lightning strike detected in the
SQL database.
Field
count
Data Type
integer
year
month
day
time
integer
integer
integer
text
timestamp
bearing
distance
integer
integer
integer
x
y
strike_type
integer
integer
text
polarity
text
tracbearing
integer
tracdistance
integer
tracx
tracy
integer
integer
Description
A unique identification given to each strike by
NexStorm, used to confirm each strike is new data.
The data of the lightning strike.
The time of the strike in the form H:mm:ss, calculated
by the Lightning Data Acquisition Tool.
The time of the strike in seconds from 0:00 midnight.
The bearing of the strike in 10th of degrees from North.
The distance of the strike in kilometres from the
detector.
Co-ordinates of the strike on a 1000x1000 pixel map
with the detector in the centre. Calculated by NexStorm.
The type of the strike: C = intra-cloud, G = cloud to
ground.
The polarity of the cloud emitting the strike (comparable
to the direction of electron flow in the strike): + =
positive, - = negative.
The bearing of the thunderstorm (determined by
NexStorm’s TRAC – Thunderstorm Ranging and
Acquisition engine) in 10th of degrees from north.
The distance, in kilometres, of the thunderstorm from
the detector.
Co-ordinates of the thunderstorm on a 1000x1000 pixel
map with the detector in the centre.
• Table 4.1: Lightning Strike Data database schema
Figure 4.1 shows the data being accessed directly from the database using the pSQL
PostgreSQL client for UNIX. A number of flexible queries can be passed to the
database server to get the data required.
14
•
Figure 4.1: Lightning strike data being accessed via the UNIX pSQL client.
Graphical Display of Data
As well as the raw data in the database, it would be useful to have graphical means of
displaying the lightning strike data.
It was decided that IDL® (the Interactive Data Language) should be used to plot the
data as this provides a powerful set of tools to build many custom charts and can
execute a precompiled set of commands to automate the plotting procedure. Data can
be imported from a number of sources. IDL® also provides a number of tools to draw
maps of the world (built-in) and superimpose plots of data on top of the map.
The data, including plots will be made available on the World Data Centre web site
and will update automatically, therefore Perl scripts were written to compile the IDL®
command file containing the data from the database to plot the data as a GIF file
which is in the web page. Perl was chosen as it allows powerful scripts to be written
quickly and it has a number of modules to add features such as PostgreSQL database
access.
Mapping Lightning Strikes
It would be useful to have a plot showing the location of each lightning strike to quickly
see the situation. This can be achieved by calculating the longitude and latitude of
each strike from the distance and bearing of the strike and the location of the detector,
then plotting these points straight onto the map of the appropriate area.
It was decided that each plot should show the strikes that occurred within the last two
hours, using a colour code to separate the times of the strikes into twenty minute
intervals and different symbols to distinguish the type and polarity of each strike. The
date and time period of the data shown is automatically placed at the top of each plot.
15
• Figure 4.2: Map of lightning strikes within a two hour period. The symbols correspond to the
type and polarity of the strike and the colour to the time.
Plotting the Strike Rate
IDL® also allows contour plots to be produced where each line corresponds to, in this
case, the lightning strike rate (which will be calculated by the Perl script compiling the
commands). Each time a plot is produced, IDL® will automatically determine the
scaling of the contour lines and colouring will be used to distinguish them (yellow
showing the most frequent strikes). These will be plotted over the map.
• Figure 4.3: Contour plot of the lightning strike rate superimposed onto the map.
16
Calibration
5
Chapter
Calibrating the System
A number of factors can introduce inaccuracies into the data collected by the lightning
detector. The main source of inaccuracy lies in the distance readings where objects
(particularly metal) attenuate the signal causing the lightning strike to appear further
away than it actually is (as the distance is determined from the amplitude of the
signal). The system must take account of this for the data collected to be reliable.
NexStorm allows the strike ranging to be corrected either by applying a global
adjustment to the data or by only adjusting data of strikes in particular regions.
To accurately calibrate the system, data must be collected over a long period of time
and compared with other sources of lightning strike data which is available from a
number of locations:
The Met. Office
The United Kingdom Met. Office sells lightning strike data they have collected from
their detectors and are considered a reliable source of accurate information.
Other Lightning Detection Systems
A number of other lightning detection systems have been set up and some including
Island Stock (http://www.islandstock.com/weather), based in Jersey make the maps
available on the internet which can be compared with the ones produced by this
detection system. However it is unknown how reliable such data is.
• Figure 5.1: Lightning Strike Data from the Island Stock website, based in Jersey
17
Nimrod Rain Radar Data
Thunderstorms are generally accompanied by heavy rain, therefore the locations of
the lightning strikes detected can be compared with the locations of heavy rainfall
detected by the Nimrod rain radar system to which the British Atmospheric Data
Centre at RAL have access.
• Figure 5.2: Rainfall data from the Nimrod rain radar system
18
6
Chapter
Conclusions
Achievements and Future of the Project
During this project, a lightning strike data collection system was designed and
incorporated into the existing computing infrastructure in the World Data and it is in
such a state that data can be collected and used.
Before the data collected can be considered sufficiently reliable to be used for
research, the system must be properly calibrated to ensure reliability of the data. This
will require a lot of data to be collected over time and compared with data collected
from other systems.
Once the system has been properly calibrated, the research can begin into how
electrical activity in the form of lightning strikes affects the ionosphere using the data
from the system as well as the graphical representations produced. The data is stored
in the database in such a flexible way that the data can be analysed in ways other
than those produced by the scripts already written and these other methods can be
created when they are needed.
19
Appendices
i.
Flow of Data through System .............................................................21
ii.
Software Data Acquisition System Flowchart....................................22
iii.
C++ Code for Data Acquisition Tool ..................................................23
iv.
Acknowledgements.............................................................................31
20
Appendix i: Flow of Data through System
StromTracker Computer
StormTracker
Lightning
Detector
raw data
NexStorm
Lightning Data
Collection
Software
strike data
Shared System
Memory
read from memory
formatted data output
Lightning Data
Acquisition Tool
formatted data output
C1
Cluster
Comma separated
text file
PostgreSQL
Database
strike data
strike data
strike data
pSQL Client to
Access Data
WDCC1 Web Server
Strike Map
Perl Script
(CRON Job)
IDL commands
with data
idl_sdplot001.pro
IDL Command
FIle
Strike Rate Map
Perl Script
(CRON Job)
execute
execute
IDL
IDL
plot
plot
GIF Image
Plot
GIF Image
Plot
image view
image view
HTML Page
Showing Plot
HTML Page
Showing Plot
21
IDL commands
with data
idl_srplot001.pro
IDL Command
FIle
Appendix ii: Software Data Acquisition System Flowchart
User Starts Data
Acquisition
Read options from
dialog box and
store in structure.
Disable options.
Attempt to connect yes
to server then
close connection
Connected to
server OK?
yes
Data acquisition
ends
Output data to
SQL?
no
Map view of
NexStorm’s
shared memory
space for read
access
Re-enable options
in dialog box
no
no
Abort data
acquisition and
inform user
Output data to
text file?
yes
Close text output
file
Open text output
file get result
yes
Is text output
file open?
no
Create semaphore
and wait for it to
give access to
shared memory
yes
Close the
connection to the
SQL server
File opened
OK?
no
no
Send all the
commands in a
single transaction
to the SQL server
Abort data
acquisition and
inform user
Is the
semaphore
open?
End of data write
thread
yes
Read string
containing data
from memory
Copy SQL
command buffer to
temporary buffer
and clear SQL
command buffer
Was the
connection
successful?
no
try max. 10
times
Is the data noise, a
heartbeat or the same as
the previous?
yes
Open connection
to SQL server and
get result of action
yes
Print the data,
separated by
commas, into the
text file
Output data to SQL
and commands remain
in buffer?
no
yes
Copy SQL
command buffer to
temporary buffer
and clear SQL
command buffer
no
yes
Open connection
to SQL server and
get result of action
yes
no
Add the SQL
command to add
the data to the
table to the SQL
command buffer
no
try max. 10
times
Send all the
commands in a
single transaction
to the SQL server
Interpret time, type
and polarity and
put into
appropriate
formats
Output data to
text file?
Was the
connection
successful?
Close the
connection to the
SQL server
Separate data
contained in string
and store in
separate variables
yes
yes
Output data to
SQL?
Has the user
clicked the STOP
button?
no
no
no
no
Copy current data
object to previous
data object for
comparison
yes
22
Output data to
SQL?
yes
Has the loop been
round 6000 since last write
or is the SQL command
buffer full
c:\Documents and Settings\d.r.wilkins\My ...\Visual Studio Projects\Ldat\acquireData.cpp
//
//
//
//
//
//
//
//
//
//
/*
1
acquireData.cpp
===============
Code to obtain data from NexStorm via FlashGate IPC
Part of LDAT - by Daniel Wilkins
Last Edited 19/08/2004 14:35 by D Wilkins
Working release compiled 19/08/2004 14:00 by D Wilkins
(C) Copyright 2004 D Wilkins and Ionosondes Group - CCLRC Rutherford Appleton
Laboratory
PostgreSQL access functions by PostgreSQL group - http://www.postgresql.org
This program is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation; either version 2 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program; if not, write to the Free Software
Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307
USA
*/
#include "stdafx.h"
////////////////////// IPC CONFIGURATION TO GET NEXSTORM DATA ///////////////////////////
// change buffer settings below
static const long bufferSize = 100;
static const long buffers = 1;
// change name of shared memory area below
static const char memoryName[] = "NXFGIPC_SHMEM_0822931589443_238731_GATE0";
static char (*memory)[buffers][bufferSize];
static const char ReaderSemaphoreName[] = "Reader Semaphore";
static HANDLE hReaderSemaphore;
static HANDLE hMemory;
/////////////////////////////////////////////////////////////////////////////////////////
// **** FUNCTION DECLARATIONS ****
void logToFile(char[1024]);
void writeSql();
UINT thread_GetData(LPVOID);
void startthread_GetData(int*, /*CONFIGURATION:*/unsigned int, bool, char, bool, char, char
, char, char, bool, char, char, bool);
// **** GLOBAL VARIABLE DECLARATIONS ****
// stores pointer to exit signal
int* exitSignal;
// temporary string for concatenating log file entries
char logString[1024];
// for text file output
FILE* outFile;
bool fileIsOpen = false;
// signal to ensure file is open before attempting to write to
it
// for SQL buffering subsystem
vector <string> sqlBuffer(6050);
// use vectors of strings so system can manage memory
vector <string> tmpSqlBuffer(6050);
int sqlBufferCount = 0;
int tmpSqlBufferCount = 0;
///////////////////////// STRUCTURE TO HOLD SETUP DATA //////////////////////////////////
struct Config
{
unsigned int interval;
bool txtOut, sqlOut, sqlLogon, log;
char txtFile[512], sqlTable[50], logFile[512];
string sqlConnSet;
} setup;
// define structure variable to hold configuration data
///////////////////////// CLASS FOR STRIKE DATA OBJECTS//////////////////////////////////
class StrikeData
/**********************************************************************/
/* Class for lightning strike data objects.
*/
c:\Documents and Settings\d.r.wilkins\My ...\Visual Studio Projects\Ldat\acquireData.cpp
/* Holds all the data in private data items except count and year.
*/
/* HOLD DATA ITEMS
*/
/* - count (unique id given to each strike by NexStorm)
*/
/* - year, month, day of the strike
*/
/* - timestamp of the srtike (in seconds)
*/
/* - Distance, bearing, x-co-ordinate and y-co-ordinate of strike
*/
/*
both raw values and those corrected by NexStorm's TRAC system
*/
/* - strike type (intercloud or cloud-ground)
*/
/* - strike polarity
*/
/* PUBLIC FUNCTIONS TO:
*/
/* - check if strike is noise - bool isNoise()
*/
/* - check if strike is a heartbeat from NexStorm - bool isHeartbeat()*/
/* - check data is valid - bool isValidStrike()
*/
/* - output data to a text file - void toFile(bool init)
*/
/* - output data to SQL buffer to be written later
*/
/**********************************************************************/
{
public:
// public so other functions can see if it is a new strike
unsigned int count;
private:
unsigned int year,
month,
day,
timestamp;
char time[8]; // holds time in format H:mm:ss
int tracBearing,
tracDistance,
rawBearing,
rawDistance,
tracX,
tracY,
rawX,
rawY,
rawStrikeType,
rawStrikePolarity;
char strikeType;
char strikePolarity;
public:
void getData()
/****************************************************************/
/* Obtains data from FlashGate IPC (NexStorm's shared memory). */
/* Stores data in the object.
*/
/* Takes no arguments and returns no value.
*/
/****************************************************************/
{
string recvData;
DWORD semaphoreStatus;
// create read semaphore to control access to the memory - protects data structure
hReaderSemaphore = CreateSemaphore(NULL, 0, buffers, ReaderSemaphoreName);
//-------------ERROR HANDLING-----------//
if(hReaderSemaphore == NULL)
{
logToFile("Error creating read semaphore. Could not get access to NexStorm
's IPC memory.");
return;
}
// wait time set as interval for state signal from semaphore object
semaphoreStatus = WaitForSingleObject(hReaderSemaphore, setup.interval);
// proceed only if semaphore allows it
if(semaphoreStatus == 0)
{
// read in data from memory
recvData = (*memory)[0];
// put received data from string into the separate variables
2
c:\Documents and Settings\d.r.wilkins\My ...\Visual Studio Projects\Ldat\acquireData.cpp
sscanf(recvData.c_str(), "%u,%u,%u,%u,%u,%d,%d,%d,%d,%d,%d,%d,%d,%d,%d",
&count,
&year,
&month,
&day,
&timestamp,
&tracBearing,
&tracDistance,
&rawBearing,
&rawDistance,
&tracX,
&tracY,
&rawX,
&rawY,
&rawStrikeType,
&rawStrikePolarity);
// interpret strike type and polarity to display as symbol
strikeType = (rawStrikeType == 0) ? 'G' : 'C';
strikePolarity = (rawStrikePolarity == 0) ? '+' : '-';
// put time into correct format
getTime();
}
ReleaseSemaphore(hReaderSemaphore, 1, NULL);
}
void getTime()
/****************************************************************/
/* Gets the time in the format H:mm:ss from the raw timestamp
*/
/* in seconds and stores it into the object's time structure.
*/
/* Takes no arguments and returns no value.
*/
/****************************************************************/
{
int hr, min, sec;
min = timestamp / 60; // divide by 60 to find minutes
sec = timestamp % 60; // seconds is equal to the remainder
hr = min / 60;
min = min % 60;
char outMin[5], outSec[5];
// ensure values are double-digit for minutes and seconds
if(min < 10)
sprintf(outMin,"0%d",min);
else
sprintf(outMin,"%d",min);
if(sec < 10)
sprintf(outSec,"0%d",sec);
else
sprintf(outSec,"%d",sec);
sprintf(time,"%d:%s:%s", hr, outMin, outSec);
}
bool isNoise()
/****************************************************************/
/* Checks if the received strike data is 'noise'.
*/
/* If signal is noise, NexStorm sets distance and bearing to -1 */
/* Takes no arguments
*/
/* Returns BOOLEAN value true if it is noise, false if not.
*/
/****************************************************************/
{
if(rawDistance == -1 && rawBearing == -1)
return true;
else
return false;
}
bool isHeartbeat()
/************************************************************************/
/* Checks if the received strike data is a 'heartbeat'
*/
/* If signal is a 'heartbeat', NexStorm sets distance and bearing to -9 */
/* Takes no arguments
*/
/* Returns BOOLEAN value true if it is a 'heartbeat', false if not.
*/
/************************************************************************/
3
c:\Documents and Settings\d.r.wilkins\My ...\Visual Studio Projects\Ldat\acquireData.cpp
4
{
if(rawDistance == -9 && rawBearing == -9)
return true;
else
return false;
}
bool isValidStrike()
/************************************************************************/
/* Checks if the received strike data is a valid
*/
/* Checks it is not noise or a heartbeat and ensures actual lightning
*/
/* data is there (and not arbitrary values in the memory).
*/
/* Returns BOOLEAN value true if it is valid data, false if not.
*/
/************************************************************************/
{
if(!isNoise() && !isHeartbeat() && (year > 1950) && (year < 4000))
return true;
else
return false;
}
void toFile()
/****************************************************************/
/* Writes the properties of the StrikeData object to text file */
/* Requires FILE* outFile to already be opened for the file to */
/* write to.
*/
/* Takes one BOOLEAN argument - initialisation of file
*/
/* - TRUE will print headers to file
*/
/* - FALSE will print the data values
*/
/* Returns no value.
*/
/****************************************************************/
{
char output[1024];
sprintf(output,"%u , %u , %u , %u , %s , %u , %d , %d , %d , %d , %c , %c , %d , %d
\n", count, year, month, day, time, timestamp, rawBearing, rawDistance, rawX, rawY,
strikeType, strikePolarity, tracBearing, tracDistance, tracX, tracY);
// print to ouput file (already opened)
fprintf(outFile,"%s",output);
}
void toSQL()
/****************************************************************/
/* Writes the data in the StrikeData object to SQL buffer
*/
/* to be written later to the SQL database by writeSql().
*/
/* Takes no arguments and returns no value.
*/
/****************************************************************/
{
char sqlCommand[512];
sprintf(sqlCommand,"INSERT INTO %s VALUES (%u,%u,%u,%u,\'%s\',%u,%d,%d,%d,%d,\'%c\'
,\'%c\',%d,%d,%d,%d)", setup.sqlTable, /*STRIKE DATA:*/count, year, month, day, time,
timestamp, rawBearing, rawDistance, rawX, rawY, strikeType, strikePolarity, tracBearing
, tracDistance, tracX, tracY);
sqlBuffer[sqlBufferCount] = sqlCommand;
sqlBufferCount++;
}
};
////////////////////////////////////////////////////////////////////////////////////////
void logToFile(char text[1024])
/****************************************************************/
/* Passed a string which is logged to the text file (specified */
/* in the setup structure if set up to log) along with the
*/
/* current date and time.
*/
/* Returns no value.
*/
/****************************************************************/
{
if(setup.log == true)
{
FILE* logFile = fopen(setup.logFile,"a");
time_t rawtime;
c:\Documents and Settings\d.r.wilkins\My ...\Visual Studio Projects\Ldat\acquireData.cpp
struct tm * timeinfo;
// get time from system
time(&rawtime);
timeinfo = localtime(&rawtime);
fprintf(logFile, "%s: %s\n", asctime(timeinfo), text);
fclose(logFile);
}
}
UINT writeSql(LPVOID pParam)
/****************************************************************/
/* Connects to SQL database using settings in the setup struct. */
/* Then writes the contents of the temporary (working) SQL
*/
/* command buffer (which was copied from the buffer) to the SQL */
/* database.
/* Takes one argument argument - NULL - needed for threading.
*/
/* Returns 0 after successful write, 1 if connection failed.
*/
/****************************************************************/
{
PGconn* sqlConn;
// connect to PostgreSQL database
for(int connectCount = 0; connectCount < 10; connectCount++)
{
sqlConn = PQconnectdb(setup.sqlConnSet.c_str());
//--------------------- ERROR HANDLING -----------------------//
if(PQstatus(sqlConn) != CONNECTION_OK)
{
logToFile("Failed to connect to SQL server. Will try again.");
if(connectCount == 10)
{
logToFile("Giving up connecting to SQL.");
return 1;
}
}
else
break;
}
PQexec(sqlConn, "BEGIN");
for(int bufferIndex = 0; bufferIndex <= tmpSqlBufferCount; bufferIndex++)
{
PQexec(sqlConn, tmpSqlBuffer[bufferIndex].c_str()); //CHANGED FROM TEMP BUFFER
}
PQexec(sqlConn, "COMMIT");
PQfinish(sqlConn);
tmpSqlBufferCount = 0;
return 0;
}
UINT thread_GetData(LPVOID pParam)
/****************************************************************/
/* Function for data acquisition thread
*/
/* Opens text file if necessary then calls the member function */
/* to get the strike data from NexStorm's shared memory every. */
/* so often. Then closes file.
*/
/* Returns 0 if it completed successfully, 1 if it fails.
*/
/* Argument does nothing -required by MFC start thread function.*/
/****************************************************************/
{
// open the file mapping object for the shared memory space
hMemory = CreateFileMapping((HANDLE)0xFFFFFFFF, NULL, PAGE_READWRITE, 0, sizeof(char
[buffers][bufferSize]), memoryName);
// map view of the file into LDAT's address space so it can be accessed
memory = (char (*)[buffers][bufferSize])MapViewOfFile(hMemory, FILE_MAP_WRITE, 0, 0,
sizeof(char [buffers][bufferSize]));
StrikeData currentStrike;
// must retain its value when function returns to check next strike
static StrikeData previousStrike;
5
c:\Documents and Settings\d.r.wilkins\My ...\Visual Studio Projects\Ldat\acquireData.cpp
6
// open text file if necessary
if(setup.txtOut == true)
{
// open output file for append
sprintf(logString,"Opening %s for data output.",setup.txtFile);
logToFile(logString);
outFile = fopen(setup.txtFile, "a");
fileIsOpen = true;
//--------------------- ERROR HANDLING -----------------------//
if(outFile == NULL)
{
MessageBox(NULL,"The specified output file could not be opened. Please ensure a
valid filename was entered (if the file does not exist it will be created) and that it
is not in use by another process or user.\n\nData acquisition will not continue. Press
STOP to reset.", "Lightning Data Acquisition Tool", MB_ICONERROR);
logToFile("Failed to open file for output.");
fileIsOpen = false;
logToFile("Data acquisition ABORTED due to error.\n============");
return 1;
}
fprintf(outFile,"Count , Year , Month , Day , Time , Timestamp , Bearing , Distance
, X , Y , Strike Type , Strike Polarity , TRAC Bearing , TRAC Distance, TRAC X , TRAC
Y\n====================================================================================
======================================================================\n");
}
logToFile("Starting data acquisition.");
for(int acquireCount = 0; *exitSignal == 0; acquireCount++)
{
// call function to get strike data from shared memory
currentStrike.getData();
if(currentStrike.isValidStrike() && (currentStrike.count != previousStrike.count))
//check it is valid data and not a duplicate
{
previousStrike = currentStrike;
if(fileIsOpen == true)
// ensure file is actually open before writing
currentStrike.toFile();
if(setup.sqlOut == true)
currentStrike.toSQL();
}
// write contents of buffer to SQL if necessary
if(setup.sqlOut == true && (/*Time in 10ms:*/acquireCount >= 6000 || sqlBufferCount
>= 6000) && sqlBufferCount > 0)
{
// copy SQL buffer to temp working buffer
tmpSqlBuffer = sqlBuffer;
tmpSqlBufferCount = sqlBufferCount;
acquireCount = 0;
sqlBufferCount = 0;
AfxBeginThread(writeSql, NULL);
}
// wait before running it again
Sleep(setup.interval);
}
// close text file if necessary
if(fileIsOpen == true)
{
fclose(outFile);
fileIsOpen = false;
logToFile("Closed output file.");
}
// write any remaining data to databse and disconnect if necessary
if(setup.sqlOut == true && sqlBufferCount > 0)
{
tmpSqlBuffer = sqlBuffer;
tmpSqlBufferCount = sqlBufferCount;
acquireCount = 0;
sqlBufferCount = 0;
writeSql(NULL);
}
logToFile("Data acquisition stopped by user.\n============");
return 0;
c:\Documents and Settings\d.r.wilkins\My ...\Visual Studio Projects\Ldat\acquireData.cpp
7
}
void startthread_GetData(int* exitPtr, /*CONFIGURATION:*/unsigned int interval, bool txtOut
, char txtFile[512], bool sqlOut, char sqlServer[50], char sqlPort[5], char sqlDBName
[50], char sqlTable[50], bool sqlLogon, char sqlUsername[50], char sqlPassword[50],
bool log, char logFile[512])
/****************************************************************/
/* Function to start data acquisition thread
*/
/* Uses UINT thread_GetData(LPVOID pParam) function.
*/
/* Puts the configuration into the config structure
*/
/* Passed 13 arguments
*/
/* - int* for pointer to exit signal variable
*/
/* - 12 variables for program configuration
*/
/* Returns no value.
*/
/****************************************************************/
{
exitSignal = exitPtr;
//receives pointer to the exit signal and makes it accessible
to functions in this file
// get configuration from arguments and store in structure
setup.interval = interval;
setup.txtOut = txtOut;
sprintf(setup.txtFile,"%s",txtFile);
setup.sqlOut = sqlOut;
sprintf(setup.sqlTable,"%s",sqlTable);
setup.log = log;
sprintf(setup.logFile, logFile);
// check log file
if(setup.log == true)
{
FILE* testLogFile = fopen(setup.logFile,"a");
if(testLogFile == NULL)
{
MessageBox(NULL,"The specified log file could not be opened. Please ensure a
valid filename was entered (if the file does not exist it will be created) and that it
is not in use by another process or user.\n\nData acquisition will not continue. Press
STOP to reset.", "Lightning Data Acquisition Tool", MB_ICONERROR);
return;
}
fclose(testLogFile);
}
sprintf(logString,"Data acquisition started by user - Interval = %dms",setup.interval);
logToFile(logString);
// check connection to SQL server if necessary
if(setup.sqlOut == true)
{
if(sqlLogon == true)
{
setup.sqlConnSet = string("host=") + sqlServer + " port=" + sqlPort + " dbname=
" + sqlDBName + " user=" + sqlUsername + " password=" + sqlPassword + " connect_timeout
=10";
sprintf(logString,"Attempting to connect to SQL database %s on port %s of
server %s as user %s.", sqlDBName, sqlPort, sqlServer, sqlUsername);
}
else
{
setup.sqlConnSet = string("host=") + sqlServer + " port=" + sqlPort + " dbname=
" + sqlDBName + " connect_timeout=10";
sprintf(logString,"Attempting to connect to SQL database %s on port %s of
server %s.", sqlDBName, sqlPort, sqlServer);
}
logToFile(logString);
PGconn* sqlConn;
sqlConn = PQconnectdb(setup.sqlConnSet.c_str());
//--------------------- ERROR HANDLING -----------------------//
if(PQstatus(sqlConn) != CONNECTION_OK)
{
MessageBox(NULL,"An error occurred while attempting to connecting to the
PostgreSQL server. Please verify the connection settings and ensure you have access to
the specified database.\n\nData acquisition will not continue. Press STOP to reset.",
"Lightning Data Acquisition Tool", MB_ICONERROR);
c:\Documents and Settings\d.r.wilkins\My ...\Visual Studio Projects\Ldat\acquireData.cpp
logToFile("Failed to connect to SQL server.");
logToFile("Data acquisition ABORTED due to error.\n============");
return;
}
logToFile("Connected successfully to SQL server.");
PQfinish(sqlConn);
}
// start the timer function in a new thread (send no arguments)
AfxBeginThread(thread_GetData, NULL);
}
8
iv Acknowledgements
Appendix
Thanks must go to all those who have helped with this project:
•
Chris Davis for providing and running the project for the Nuffield Science
Bursary Scheme and the Nuffield Science Bursary scheme organisers
including regional co-ordinator Bridget Holligan.
•
The ionosondes group at CCLRC Rutherford Appleton Laboratory including
Sarah James, Richard Henwood and Richard Stamper for their help.
•
Matthew Wild for computing support.
•
Ionosondes group engineer John Smith and RAL Ground Station Engineer
John Wright for installing the lightning detector.
•
David Hooper in the RAL British Atmospheric Data Centre (BADC) for Nimrod
Rain Radar data.
Parts/Systems Used in this Project
•
StormTracker™ lightning detector from Boltek®: http://www.boltek.com
•
NexStorm™ lightning data acquisition software for use with Boltek® lightning
detectors and example FlashGate IPC client code made by Astrogenic
Systems: http://www.astrogenic.com
•
NSList NexStorm FlashGate IPC client from Lite-Detect project page:
http://hp.Vector.co.jp/authors/VA034934/Lightning/
•
C++ source code written and compiled using Microsoft Visual C++ as part of
Microsoft® Visual Studio® .NET 2003 Professional:
http://msdn.microsoft.com/vstudio.
•
Plots produced using IDL® from RSI®, Research Systems Inc.:
http://www.rsinc.com
•
PostgreSQL database system and pSQL UNIX command line client by the
PostgreSQL group: http://www.postgresql.org
Acknowledgements for Content of this Report
•
Information on Ionosphere (Chapter 1, page 3) adapted from “Ionosphere”,
Microsoft Encarta 2000 and information about ionospheric layers and
monitoring systems from the RAL Ionosondes group web page:
http://www.wdc.rl.ac.uk/ionosondes as well as discussion with Chris Davis.
•
Figures 2.1 and 2.2, page 5 and figure 2.4, page 7 from the Boltek website:
http://www.boltek.com
•
Figure 2.5, page 7 from the Aninoquisi Lightning/2000 page:
http://www.aninoquisi.com/lightning2000.htm
•
Screenshot (Figure 5.1) on page 17 taken from Island Stock, Jersey:
http://www.islandstock.com/weather
31