nsa 100sx

Transcription

nsa 100sx

Communications Technology
Seminar
KEPCO Utility March 13, 2006
Presented by: Angelo Rizzo and
Roger Moore
Copyright RuggedCom Inc.
1
Morning Agenda
1. Introduction to RuggedCom
2. Environmental requirements for network
equipment in Substation
3. Ethernet Essentials
4. Fiber Optic Basics
5. Substation Communication Architectures
6. Advanced Layer 2 Networking (Managed
Switches)
7. RuggedSwitch™ & RuggedServer™ Product
Overview
2
Afternoon Agenda
8. Overview of Layer 3 Networking (Routers)
9. Network Security Overview
10. Network Security NERC1300 and CIP
requirements
11. How Switches and Routers Provide Security
12. RuggedRouter™ Product Overview
13. ROS™ and ROX™ Security Features
14. Question and Answer Session
3
Introduction to RuggedCom
4
Corporate Background
A Brief History …
¾ Founded in March 2001 – Concord, Ontario, Canada
¾ Primary objective was to develop “Substation Hardened” Fiber Optical Ethernet
Switches and Routers
¾ Currently employ over 50 people: Engineering, Manufacturing, Sales and Support
¾ Key Utility Investors: Ontario Power Generation, EPCOR Utilities
Related Industry Background …
¾ Over 100 years of collective experience in the design of Protective
Relaying Systems, Industrial Automation and Substation Automation
technology
¾ Over 100 years of collective experience in the design of
Communications Systems and Networks
Leverage Our Combined Experience in Automation and
Communications to Get the Best of Both Worlds
5
Facilities
Corporate Headquarters
¾ Woodbridge (Toronto), ON,
Canada
¾ R&D, Sales and Marketing, Final
Assembly, and 24 hr Burn-in
¾ Technical Support and Training
Center
Manufacturing
Customer Support Center
¾ Creation Technologies - Toronto
¾ Hollywood Florida
¾ Local contract manufacturer
¾ Staffed by Certified RuggedCom
Network Engineers
¾ New state-of-the-art SMT facility
¾ ISO9001 – 2000 certified
¾ Six-Sigma methodologies &
metrics
¾ Customer Support
¾ Equipment for Proof of Concept
¾ Technical Training
¾ Interoperability Testing
6
Markets Served
Electric Power Utilities - Substation Automation
¾ “Substation Hardened” Ethernet is emerging trend for substation
automation (UCA, IEC 61850, IEEE1613)
¾ Critical element for a smart, self-healing, power grid
Industrial – Process Control & Plant Floor Automation
¾ “Industrially Hardened” Ethernet on the plant floor is new trend for
plant/process control.
¾ Over 6.0 Million Ethernet devices/year expected to be shipped by 2007
Intelligent Transportation Systems (ITS)
¾ “Field Hardened” Ethernet is new emerging trend for deployment of video
over IP & traffic control at major intersections.
¾Train control systems / light rail converting to Ethernet
Government/Military
¾ “Environmentally Hardened” Ethernet is new emerging trend for use in
military
¾ Homeland security, and government infrastructure (e.g. Pentagon)
7
Leadership In The Industry
¾ 1st to implement Ethernet and UCA in a protective relay
- Company founders were pioneers in integrating Ethernet and UCA in protection relays
¾ 1st and only Ethernet switch to provide Zero-Packet-Loss™
- performance under high levels of electromagnetic interference
- IEC 61850-3 (2002) requirements for communications networks in substations
¾ 1st and only Ethernet switch qualified as an IEEE 1613 Class 2 error free communications
- for communications networks in electric power substations
- strong participation in the development of the IEEE 1613 (2003) standard
¾ 1st and only Ethernet switch to offer Enhanced RSTP (eRSTP)™
- for fault tolerant ring architectures with high-speed fault recovery of < 5ms per hop
¾ 1st and only Ethernet switch to achieve UL864 approval
- for use in fire control systems for the Pentagon.
¾ 1st and only Managed Ethernet switch that is IP65/IP67 rated
- Waterproof: IP65 (Water Jets) and IP67 (Water Immersion)
Building On Our Heritage as Innovators in Substation Automation
8
Focus on Quality
Quality Management System (QMS)
¾ Quality manual, processes and auditing in
place
¾ Dedicated QA manager
¾ Formal quarterly metrics analysis and
management review.
¾ ISO9001:2000 certification
Six-Sigma Methodologies Employed
¾ Key manufacturing and engineering personnel
have 6-sigma background and training.
¾ DMAIC applied in manufacturing: FPY, Pareto,
Defect
Analysis
9
Product Basket
RuggedRouter™ Routers
¾ RX1000 – Integrated Router, Firewall, and VPN
¾ RX1100 – RX1000 features plus IDS and BGP (coming soon)
Gigabit RuggedSwitch™ Ethernet Switches
¾ RSG2200 – 9-Port Managed Gigabit Ethernet Switch
¾ RSG2100 – 19-Port Modular Ethernet Switch with Gigabit Options
¾ RS900G – 10-Port Flexible Ethernet Switch with Gigabit Options
¾ RS969 – IP65/IP67 Rated 10-Port Ethernet Switch with Gigabit Options
RuggedSwitch™ Ethernet Switches
¾ RS900 – 9 Port Ethernet Switch with Fiber Optic Options
Serial Servers and Media Converters
¾ RS400 – 4 Port Serial Device Server with 4 Port Ethernet Switch
¾ RMC – Ethernet Media Converter (Copper to Fiber)
¾ RMC20 – Serial Media Converter (Copper to Fiber)
¾ RMC30 – 2 Port Serial to Ethernet Converter
¾ RMC40 – 4 Port Ethernet Media and Speed Converter
Most Complete Line of Substation Hardened Communications Devices
10
Environmental Requirements for
Network Equipment in a Substation
11
Substation Environment
Power Station
MV/HV Substation
EMI & Environmental Phenomena Typical of Substation Environments
• Electric and Magnetic Fields
• Electrostatic Discharge
• Conducted High Frequency Electrical Transients
• High Energy Power Surges
• Ground Potential Rise during ground faults
• Climactic Variation: Temperature & Humidity
• Seismic / Vibration
• Pollution: Dust, Metallic Particles, Condensation, Solar Radiation
12
Substation EMI Phenomenom
Continuous
Phenomena
•• Radiated
Radiated RFI
RFI
•• Induced
Induced RFI
RFI
•• Power
Power freq.
freq. Magnetic
Magnetic Field
Field
•• Slow
Slow Voltage
Voltage Variations
Variations
•• Harmonics,
Harmonics, Interharmonics
Interharmonics
•• Ripple
Ripple on
on d.c.
d.c. power
power
supply
supply
•• Power
Power Frequency
Frequency Voltage
Voltage
Transient
Phenomena
(High Occurrence)
•• Electrostatic
Discharge
Electrostatic
Discharge
Transient
Phenomena
(LowFrequency
Occurrence)
•• Power
Power
Frequency
•• Voltage
Voltage Dips
Dips
•• Lightning
Lightning
•• HV
HV Switching
Switching by
by Isolators
Isolators
•• Reactive
Reactive Load
Load Switching
Switching
Variation
Variation
•• Power
Power System
System Faults
Faults
•• Short
Short Duration
Duration Power
Power
Freq.
Freq. Magnetic
Magnetic Fields
Fields
Devices in substation environments must deal with
a combination of EMI phenomena which are both
continuous and transient.
13
Communication Standards
IEC 61850-3 (2002)
IEEE 1613 (2003)
“Communications networks and
systems in substations”
“Standard Environmental and Testing
Requirements for Communications
Networking Devices in Electric Power
Substations”
14
IEC 61850-3: EMI Immunity
UTILITY IEC 61850-3 (61000-6-5) Communications Networks and Systems In Substations (Jan 2002)
Description
TEST
IEC 61000-4-2
ESD
IEC 61000-4-3
Radiated RFI
IEC 61000-4-4
Burst (Fast Transient)
IEC 61000-4-5
Surge
IEC 61000-4-6
Induced (Conducted) RFI
IEC 61000-4-8
Magnetic Field
IEC 61000-4-29
Voltage Dips & Interrupts
IEC 61000-4-11
IEC 61000-4-12
Damped Oscillatory
IEC 61000-4-16
Mains Frequency Voltage
IEC 61000-4-17
Ripple on D.C. Power Supply
Enclosure Contact
Enclosure Air
Enclosure ports
Signal ports
D.C. Power ports
A.C. Power ports
3
Earth ground ports
Signal ports
D.C. Power ports
A.C. Power ports
Signal ports
D.C Power ports
A.C. Power ports
3
Earth ground ports
Enclosure ports
D.C. Power ports
A.C. Power ports
Signal ports
D.C. Power ports
A.C. Power ports
Signal ports
D.C. Power ports
D.C. Power ports
Test Levels
Severity Levels
+/- 6kV
3
+/- 8kV
3
10 V/m
3
+/- 4kV @ 2.5kHz
x
+/- 4kV
4
+/- 4kV
4
+/- 4kV
4
+/- 4kV line-to-earth, +/- 2kV line-to-line
4
3
4
10V
3
10V
3
10V
3
10V
3
40 A/m continuous, 1000 A/m for 1 s
N/A
30% for 0.1s, 60% for 0.1s, 100% for 0.05s
N/A
30% for 1 period, 60% for 50 periods
N/A
2
100% for 5 periods, 100% for 50 periods
N/A
3
2.5kV common, 1kV differential mode @ 1MHz
3
3
30V Continous, 300V for 1s
4
30V Continous, 300V for 1s
4
10%
3
9 EMI Immunity Requirements based on Substation Environment
9 Higher EMI Immunity Levels than Industrial Environments
9 Similar type tests as per Protective Relaying IEDs
15
IEC 61850-3: Performance
IEC 61000-6-5 Table 7 - Performance Criteria for the most relevant functions
(in descending order of criticality)
Functional requirements versus electromagnetic phenomena
Functions
Continous phenomena
Transient phenomena
with high occurance
Transient phenomena
with low occurance
Protection and teleprotection
On-line processing and
regulation
Metering
Short delay
Command and Control
Supervision
** No Delays or Data Loss **
Stop and reset
Man-machine interface
Alarm
Data transmission and
telecommunication
Temporary loss, self
recovered
Short delay, temporary wrong indication
No loss, possible bit
error rate degradation
Temporary loss
Temporary degradation
Data acquisition and storage
Temporary degradation, self recovered
Measurement
Off-line processing
Temporary degradation Temporary loss and reset
Passive monitoring
Temporary degradation
Temporary loss
Temporary loss, self recovered
Self-diagnosis
16
IEC 61850-3 Climatic Specs
Four Classes of Locations:
1.Class A: air-conditioned locations (indoor)
2.Class B: heated or cooled enclosed conditions
3.Class C: sheltered locations
4.Class D: outdoor locations
Class C Operating Temperature Ranges:
1.Class C1: -5 to +45°C
2.Class C2: -25 to +55°C
3.Class C3: -40 to +70°C
4.Class Cx: Special (defined by mfg)
17
IEEE 1613: EMI Immunity
IEEE P1613 ? Draft Standard Environmental Requirements for Communications Devices Installed in Electric
Power Substations
TEST
Description
IEEE C37.90.3
ESD
IEEE C37.90.2
Radiated RFI
IEEE C37.90.1
Fast Transient
IEEE C37.90.1
Oscillatory
IEEE C37.90
Dielectric Strength
Test Levels
Enclosure Contact
Enclosure Air
Enclosure ports
Signal ports
D.C. Power ports
A.C. Power ports
3
Earth ground ports
Signal ports
D.C. Power ports
A.C. Power ports
Signal ports
D.C. Power ports
A.C. Power ports
+/- 8kV
+/- 15kV
35 V/m
+/- 4kV @ 2.5kHz
+/- 4kV
+/- 4kV
+/- 4kV
2.5kV common mode @ 1MHz
2.5kV common & differential mode @ 1MHz
2.5kV common & differential mode @ 1MHz
2kVac
2kVac
2kVac
Severity Levels
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
9 Based on IEEE C37.90.x standards for Protective Relaying devices
9 Two Performance Classes of Device Operation
CLASS 1 – communications errors allowed during type tests
CLASS 2 - “error free” operation during type tests
9 No cooling fans allowed!
18
IEEE 1613 Class 2 Performance
Power System
Current Waveform
Fault
Substation LAN
τ = 16.7ms
SWITCH
GOOSE
Message
SWITCH
Protective
Relaying IEDs
Fault period is a period of:
•
•
•
•
•
High levels of transient EMI phenomena!
Sub-cycle (i.e. ≤ 16.67ms) processing by IEDs
Substation LAN could be flooded with control traffic e.g.
GOOSE/GSSE messages for tripping/blocking!
LAN must perform without communications errors, or delays due to
EMI conditions caused during fault period!
IEEE 1613 Class 2 networking devices required for LAN!
19
IEEE 1613 Climatic Specs
Device Operating Temperature Ranges
a) -40 to +70°C
b) -30 to +65°C
c) -20 to +55°C
d) Range defined by the manufacturer
IEEE 1613 also stipulates…No cooling fans allowed!
20
Fiber vs. Copper: EPRI Study
EPRI - Study of copper Cable for UCA (1997)
• Looked at susceptibility of shielded and unshielded
CAT5 cable to electrical fast transients which are a
common EMI phenomenon in substations.
• Results indicated large communications frame loss
rates:
» 32% @ 1kV
» 66% @ 2kV
» 75% @ -2kV
21
Fiber vs. Copper: Rockwell Study
Rockwell Automation - CMR of Copper Cable (2002)
• Induced RFI (IEC 61000-4-6) Applied to CAT5 cable
to test CMR (Common Mode Rejection)
• Common mode noise coupling will occur via
adjacent cabling
• Resultant Bit Error Rate:
» 22% @ 10Vrms (noise coupled)!
CONCLUSION
Fiber is required for real-time control applications
where communications errors cannot be tolerated.
22
RuggedRated™ Specifications
¾ Rated for reliable operation in harsh electrical environments
- Electric utility substations: Meets IEEE 1613, Exceeds IEC 61850-3
- Variable speed drive systems: Exceeds IEEE 61800-3
- Generic industrial environments: Exceeds IEC 61000-6-2
- Traffic control equipment: Exceeds NEMA TS-2
¾ Rated for operation over a wide temperature range
- -40°C to +85°C (+185°F)
- Passive cooling – no fans
- CSA/UL 60950 safety approval to +85°C
¾ Rated for high availability
- Integrated single and dual redundant power supplies
- 24VDC, 48VDC, or 88-300VDC/85-264VAC
- Dual power supplies can be powered from different sources
¾ Rated for industrial installations
- 18 gauge galvanized steel enclosure for durability
- heavy duty 19” rack or din rail mount
- industrial terminal blocks for power and I/O connections
¾ 5-Year Warrantee
All RuggedCom Products are RuggedRated™
23
Zero Packet Loss Technology
Transients Generator
IEC 61850, IEEE 1613
Ingoing packets
Outgoing packets
SmartBits
(Network Simulator/Analyzer)
RuggedCom Products Meet or Exceed the Most Extensive Set
of EMI Immunity Standards in the Industry
24
Ethernet Essentials
25
Ethernet History
•
•
•
•
Invented by Robert Metcalf at Xerox
Celebrated 30th birthday 2003
Dominant in office LAN
Survived where Token Ring, FDDI, Arcnet failed
26
OSI Seven Layer Model
IEC61850(UCA2)
ProfiNet
Fieldbus HSE
Application
HTTP
EtherNet/IP
SMTP
DNP3/TCP
Modbus/TCP
FTP
…
Presentation
Session
Transport
TCP/UDP
Network
IP
Data Link
IEEE 802.1
Physical
IEEE 802.3
TCP/IP
Ethernet
Please Do Not Take Sausage Pizza Away!
27
Communication Within OSI Model
Station 1
Station 2
Application
Application
Presentation
Presentation
Session
Session
Transport
Transport
Network
Network
Link
Link
Physical
10/100 BaseTx
Physical
28
The Ethernet Frame
• Ethernet transmits data in a frame of size 64 to 1518
bytes except when VLAN tagged (more on that later)
• Frame overhead of 18 bytes (min) results in overhead
of 28% - ouch – for small data payloads
29
Ethernet Addressing
• 48-bit number
– First three bytes assigned by IEEE (Organization Unique
Identifier – OUI)
– MSBit indicates unicast or multicast
– Broadcast address is 0xFFFFFF
• All ‘MAC addresses’ for all devices ever built have a
unique address
• Source address always the senders unique address
• Destination address can be one of:
– Unicast – station to one other station
– Multicast – station to multiple stations
– Broadcast – station to all other stations
30
The Ethernet Repeater
• Also known as a hub
• Layer 1 only – does not understand Ethernet frame
• Repeats incoming signal all other ports with restored
timing and signal strength with negligible delay
• Requires CSMA/CD i.e. collisions & non-deterministic
• Half-duplex only
• Collision domain restricting network ‘diameter’ to 200m
Station 1
Station 2
Application
Presentation
Session
Transport
Network
Link
Physical
Application
Presentation
Session
Transport
Network
Link
Physical
Repeater
Physical
10/100 BaseT
31
•
CSMA/CD is History!
Ethernet on a shared media requires CSMA/CD to
allow equitable use of the media
–
–
–
•
Operation of CSMA/CD
–
–
–
•
•
CS - Carrier Sense (Is someone already talking?)
MA – Multiple Access (I hear what you hear)
CD – Collision Detection (Hey, we’re both talking!)
If the medium is idle transmit anytime
If the medium is busy wait and transmit right after
If a collision is detected, backoff for a random time and repeat
Collisions have been the historical complaint against
Ethernet – not ‘deterministic’
A fully switched Ethernet network only has point-topoint connections: no CSMA/CD and no collisions
32
The Ethernet Switch
•
•
•
•
•
•
•
Buffers frames before re-sending: “store and forward”
Checks frame integrity (CRC)
Allows full duplex links and speed conversion
No CSMA/CD – No Collisions - Deterministic
Supports flow control via PAUSE frames
Traffic queued at egress ports to eliminate collisions
Automatically learns the addresses of all end devices
Station 1
Application
Presentation
Session
Transport
Network
Link
Physical
Station 2
Switch
Link
Physical
Application
Presentation
Session
Transport
Network
Link
Physical
33
Layer 2 Switch Basic Operation
1. Error check incoming frame
2. Teach MAC table source address and ingress
port relationship
3. Lookup destination address from MAC table
to determine egress port, VLAN membership,
and CoS weighting, …
4. Put frame onto egress port queue or ‘flood’
unknown address
5. Add or remove 802.1P/Q tag based on user
configuration
6. Transmit frame
34
The Managed Switch
• Same as unmanaged but add:
– User interface via RS232, Telnet, SNMP, HTTP, …
– Status, statistics, and troubleshooting facilities
– Rapid Spanning Tree (IEEE 802.1w) for fault tolerant
loop architectures
– VLANs (802.1Q)
– Quality of Service-QOS (802.1p)
– SNMPv2, RMON Groups 1, 2, 3, 9
– IGMP(Internet Group Messaging Protocol) Snooping
– GMRP(Generic Multicast Registration Protocol)
– GVRP(Generic VLAN Registration Protocol)
– Link aggregation (IEEE 802.3ad)
– Port Mirroring
– And more …
35
Switches Add Latency
• Latency is the time is takes a frame to get from
source to destination
• Store and forward introduces a minimum
latency of one frame time per switch ‘hop’
• Frame latency ranges from 5 to 120 us per
switch at 100Mbps - proportional to frame size
• Switch also introduces a processing latency on
the order of 5us
• To get absolute worst case multiply by
maximum queue size at busy port
• QoS pushes important traffic to front to queue
to reduce latency
36
Auto Crossover, Sensing, and
Negotiation
• Auto-crossover auto-detects Rx/Tx pairs
eliminating need for crossover cables
– Doesn’t work on fiber links
• Auto-sensing auto-configures link speed
– Only works for 10/100 UTP links
– Cannot configure duplex
• Auto-negotiation auto-configures speed, duplex,
and flow control
– Doesn’t work on fiber links (except 100SX)
– Duplex mismatch problem can occur when both
sides don’t negotiate
37
Fiber Optic Basics
38
Fiber Optics Overview
•
•
•
•
•
•
•
•
Immense bandwidth
Long distances possible
Immune to electromagnetic interference
Increased security (resistance to
eavesdropping)
Future proof
Lightweight
Higher pull strength then typical copper cabling
Cost continues to drop
39
Fiber Optical Cable
• Consists of three layers:
– Core - Very thin strand of glass that carries the data
– Cladding - Another layer of glass with a different refraction
index to either keep light in or out of the core
– Buffer - protective layer
• Two types: ‘multi mode’ and ‘single mode’
• Data transmitted with a single frequency of light
40
Multimode vs. Singlemode
• Multi-mode:
– Uses graded index cladding to reflect
light back into core
– 62.5/125mm or 50/125mm
– 850/1300 nm light
– Lowest cost
– Distance limited by modal dispersion
• Single mode:
– 8/125mm or 9/125mm
– 1310/1550 nm light
– Distance limited by attenuation and
chromatic dispersion
41
Fiber Optic Connectors
•
•
•
•
ST “Stick and Twist” and SC “Stick and Click” historically popular
LC becoming prevalent especially for Gigabit because small form
factor (SFF) allows greater port density
GBIC are pluggable SC transceivers using SC connectors
SFP are ‘Small Form Factor Pluggable” using LC connectors
ST
LC
SC
MTRJ
42
IEEE Fiber Standards
Standard
Wavelength
Data Rate
Distance*
Notes
10Base-FL
850nm
10 Mbps
2 km (MM)
15 km (SM)
Fiber optic end stations
100Base-Fx
1300nm
100 Mbps
2km (MM)
15km (SM)
Typical Fast Ethernet
100Base-SX
850nm
10/100 Mbps
300m (MM)
Negotiates speed
Not popular
1000Base-SX
850nm
10/100/1000
Mbps
220m (62.5 MM)
550m (50 MM)
Negotiates speed
Not popular
1000Base-LX
1310nm
1000 Mbps
550m (MM)
5km (SM)
Typical GigE
1000Base-Bx
1310 /
1550 nm
1000 Mbps
5km (SM)
Bi-directional single fiber
strand optics
1000Base-LH
1550nm
1000 Mbps
70 km (SM)
Long-haul backbones
*Distances can be increased when using higher grade cable and high power transmitters.
43
Fiber Power Budget
Net Power Budget = Launch Power – Receiver Sensitivity – Signal Loss
Signal Loss = Attenuation of Fiber + Splice + Connector(s)
• Optical power measured in dBm – referenced
to 1mW
• Launch power and receiver sensitivity vary from
vendor to vendor
• Fiber attenuation ranges from 3dB/km for
850nm MM to 0.2 dB/km for 1550nm SM
• Splice attenuation approx 0.1 dB
• Connector attenuation approx 1.0 dB
44
Fiber Distance Limitations
• Maximum distance allowed with multi-mode
fiber is limited primarily by modal dispersion
– Modal dispersion caused by light traveling different
paths through the fiber resulting in signal ‘smudge’
– Cables rated in MHz/km
• Single-mode fiber limited by attenuation and
chromatic dispersion
– Attenuation due to impurities in the glass fiber –
measured in dM/km
– Chromatic dispersion caused by transmitter emitting
spectrum of frequencies resulting in signal ‘smudge’
– Long distances possible (>100km) with high quality
cable and high power transmitters
45
Fiber Do’s and Don’t’s
• DO Keep your fiber clean. Use an airgun or
alcohol swab to clean the ends
• DO Keep fiber cables capped when not in use
to prevent dust and scratches
• DO Keep fiber ports on devices capped when
not in use to prevent dust.
• DON’T Let the fiber bend more than a 10cm
radius. Fiber is glass and breaks. Bending also
adds attenuation.
• DON’T Look into a fiber! Use an optical power
meter.
46
Substation
Communication Architectures
47
Brief History
•
Driven by Utilities and EPRI during 90’s
•
Developed UCA2.0 – Utility Communications Architecture
specification in an attempt to provide one common protocol
and architecture for utility communications.
•
UCA2.0 profile specified Ethernet (IEEE 802.3)
•
Adoption by ALL major IED Vendors in late 90’s
•
UCA2.0 specification becomes IEC 61850 – issued 2002/3/4.
•
Other popular protocols Ethernet: DNP3, Modbus, Profibus,
DeviceNet…
48
Typical Ethernet Substation
Substation LAN
IEDs
HV Wiring
C
A
Breaker
Source Source
CTs
VTs
Power System
B
The Substation LAN provides a high-speed communications bus between
a variety of IEDs (e.g. Relays, RTUs, Meters, etc…)
49
IEC 61850 Substation
“Station Bus” LAN
IEC 61850-8-1
“Process Bus” LAN
IEC 61850-9-2
IEDs
Digital
CT/VT
IEDs
C
Intelligent
Breaker
A
Power System
B
The “Digital Substation”: both power system data and control over the LAN.
50
Cascading Bus Architecture
2
1
IED IED
IED
IED IED
IED
IED
3
IED IED
IED
IED
4
IED IED
IED
N
IED
IED IED
IED
IED
IED
• Cost-effective bus architecture – messages cascade
from switch to switch.
• Maximum number of “hops” (N) is determined by
worst case latency requirements
51
Star Architecture
IED IED
IED
IED IED
IED
IED IED
IED
IED
IED
IED IED
IED
IED
IED
• Low-Latency Architecture – Any IED to IED
communications requires only two ‘hops’.
• “Home run” cabling often more expensive or prohibitive
to implement
52
Ring Architecture
PATH
1
IED IED
IED
Fault
Fault
IED IED
IED
IED
IED IED
IED IED
IED
IED
IED
PATH
2
IED
IED IED
IED
IED
IED IED
IED
IED
IED
• N+1 Fault Tolerant Ring Architecture
• Automatic Reconfiguration via RSTP
53
RuggedCom Substation Network
54
Advanced Layer 2 Networking
(Managed Switches)
RSTP, VLAN, CoS, IGMP, 802.1x,
SNMP, RMON, …
55
Spanning Tree Protocol (802.1d)
• Allows for redundant connections by preventing loops in
the LAN*
• Automatically determines the best fit tree that spans the
entire physical LAN
• Provides fault tolerance by using redundant links as
backups
• Has low overhead, is reproducible
• Minimal configuration required (if any)
• Is an industry standard with vendor interoperability
• 30 second failover times
* The presence of ‘loops’ in an Ethernet LAN results in a
broadcast storm which renders the network useless.
56
Rapid Spanning Tree (802.1W) RSTP
•
•
•
•
•
•
•
Builds on the features of STP
Backwards compatible with STP
Introduces Link State Failover for fast recovery
Very fast failover times under 50 ms
Slow link recovery times up to 2 seconds
Ring size limited to 20 switches
Requires managed switches with RSTP
capability to capitalize on fast failover
57
RSTP Roles and States
• Switch states:
– Root bridge: the logical center of the network
– Designated bridge: not the root bridge
• Port states:
– Discarding: no address learning nor frame forwarding
– Learning: addresses learned but no frame forwarding
– Forwarding: learning and forwarding
• Port Roles:
– Root: best root to the root bridge – can only have one
– Designated: best port for servicing the LAN segment to which is
is connected
– Alternate: an alternate to the current root port
– Backup: backup for a designated port
58
RSTP Example: Steady State
L2
SW
Root bridge
1
Root Port
L2
SW
2
3
4
5
Ethernet Switches
A,B,C,D
A
6
Alternate Port
1
2
3
4
5
x
6
B
y
L2
SW
1
2
3
4
5
6
C
Backup Port
Designated Port
L2
SW
Traffic from x to y must go
through root bridge A instead
of the more direct path
1
z
2
3
4
5
6
D
Designated Port
59
RSTP Example: After Failover
L2
SW
Failed Link
L2
SW
1
2
3
4
5
x
6
1
B
2
3
4
5
Alternate link is activated
thus providing a path
back to root bridge.
A
6
L2
SW
1
2
3
4
5
6
C
L2
SW
1
2
3
4
5
6
D
y
Traffic from x to y now takes
the more direct path after the
root port failure.
z
60
eRSTP Technology
Port unblocks and
allows new path
New
Network Traffic
Path(s)
Enhanced Rapid Spanning Tree
Protocol (eRSTP™)
¾ Enhanced IEEE 802.1w RSTP
¾ High-Speed Fault Tolerant Ring Architectures
¾ Fast Fault Recovery: < 5ms/hop fault recovery
¾ Large Rings Configurations: up to 80 switches
¾ Compatible with RSTP (IEEE 802.1w)
FAULT
¾ Available on RuggedSwitch™ devices only
eRSTP™ Fastest Network Fault Recovery in the Industry …
<5ms/hop vs. 300ms of competitors
61
VLAN (802.1Q)
•
•
•
•
•
Virtual LAN: an independent Ethernet network that shares cabling
infrastructure with other networks
Allows multiple end-stations at different physical location to act as
one logical group
Each VLAN has a separate “broadcast domain”
IEEE 802.1Q standard defines ‘tagged’ frame format allowing
multiple VLANs to be carried on a ‘trunk’
‘Bridging traffic between VLANs requires a router
1
2
3
4
Ethernet Switch
192
62
VLAN: Tagged vs. Untagged
• Untagged’ frames are standard 802.1d frames
– Most end devices (PC, PLC, IED, …) send and receive
untagged traffic on what is termed an access port
• ‘Tagged’ frames contain 802.1P/Q extension
– Tagged traffic is typically only found on trunk ports in the ‘core’
of the network interconnecting switches and routers
– VLAN ID ranges from 1 to 4095; priority from 0 to 7
– Ether-type TPID field always 0x8100
6 bytes
6 bytes
2 bytes
Variable
Dest.
Src.
Length / Type
Data
6 bytes
6 bytes
2 bytes 2 bytes
Dest.
Src.
TPID
TCI
3 bits
Standard
Frame
2 bytes
Variable
Length / Type
Data
1 bit
Priority CFI
12 bits
VID
Tagged
Frame
63
VLAN: Example Network
L3
1
RT
Three VLANs:
Red, Green,
and Blue
L2
SW
1
2
3
4
5
2
3
4
R
Router or L3 Switch
needed to traverse
R,G,and B VLANS
A
6
802.1Q Trunk
Tagged Frames
L2
SW
1
2
3
4
5
6
B
L2
SW
1
2
3
4
5
6
Ethernet Switches
A,B,C,D
C
L2
SW
End Device
PC, PLC, IED, …
1
2
3
4
5
6
D
Access/Edge Port
Untagged Frames
64
VLANs: Why Bother?
• Lots of broadcast traffic wastes bandwidth
– VLANs reduce this traffic since it only goes where needed
• Isolate stations with critical real-time traffic
– Such devices won’t have the processing overhead for unrelated
traffic
• Isolate stations with excessive traffic output
– Video surveillance equipment will generate prodigious amounts
of traffic – VLANs keep it separated
• Security
– VLANs restrict traffic to required stations – can’t sniff
• Typically each VLAN given its own IP subnet
65
GVRP: Generic VLAN Registration
• An industry standard protocol for
propagating VLAN info across the LAN
• Simplifies VLAN administration in the
network core
• Allows for VLAN pruning which can save
network bandwidth
• If all end devices supported GVRP there
would be no need to statically configure
switches!
66
VLAN Without Pruning
L3
1
RT
L2
SW
1
2
3
4
L2
SW
1
2
5
6
B
3
4
5
3
4
R
A
6
L2
SW
2
1
2
3
4
5
6
C
L2
SW
1
2
3
4
5
6
D
Without GVRP pruning all trunks port carry
all VLANs all the time potentially wasting
bandwidth. Switches C and D have no need
for Green traffic at all but they still get it.
67
VLAN with GVRP Pruning
L3
1
RT
L2
SW
1
2
3
4
L2
SW
1
2
5
6
B
3
4
5
3
4
R
A
6
L2
SW
2
1
2
3
4
5
6
C
L2
SW
1
2
3
4
5
6
D
With GVRP pruning the green VLAN traffic
is restricted to switches A and B and the red
VLAN is restricted to A,C, and D. The blue
would still be carried on all the trunks.
68
Cos vs. QoS
• CoS (Class of Service) = Prioritization
– Supported by IEEE 802.1p standard
– Provides priority queuing of data packets from
source to destination
– Best effort service
• QoS (Quality of Service) = Consistency
–
–
–
–
Provide CBR-like (constant bit rate) service
Predictable latency
Bandwidth Reservation
RSVP = bandwidth reservation protocol
69
IEEE 802.1p Prioritization
•
Switch
1 2
2 1
2 2
•
2 1 1 •
•
•
Multiple egress traffic queues to that
higher priority traffic can be sent first
Time-sensitive traffic (like voice /
GOOSE) can have reduced jitter and
latency
User configurable policy for ‘weighting’
scheme that determines how egress
queues are emptied
Managed switches have ability to
classify and tag incoming untagged
traffic based on port number, address,
or DiffServ
Shares tag header with 802.1Q VLAN
70
Multicast vs. Unicast vs.
Broadcast
• Multiple unicasts are slow & inefficient
• Broadcasts span entire network: too much
processing, wasted bandwidth
• Multicasts are controllable
Clients
Video Server
71
Benefits of Multicasting
• Supports one-to-many and many-to-many
delivery
• Bandwidth savings
• Reduces processing load on hosts with no
interest in the application
• Critical for conserving expensive WAN
bandwidth
• Use IGMP or GVRP to optimize network traffic
flows
72
IGMP Snooping
• Internet Group Management Protocol ensures multicast
traffic is directed only to desired recipient.
• Producer -> consumer model
• Used by IP hosts to report their host group
memberships to multicast routers. As hosts join and
leave specific multicast groups, streams of traffic are
directed to or withheld from that host.
• IGMP protocol operates between multicast routers and
IP hosts.
• IGMP snooping protocol defined for managed switches
to perform just like an IGMP router
• A layer 3 protocol in a layer 2 switch
73
Network Before IGMP
c1
c1
Consumer of IP group
eg. Display/decoder
L2
SW
1
2
3
4
L3
1
RT
L2
SW
1
2
5
6
B
3
4
5
3
4
1
2
R
Without pruning, all stations
see the IP group traffic
eg. Z1 and z2 see the traffic
A
6
L2
SW
2
3
4
z1
5
6
C
L2
SW
1
2
3
4
5
6
D
p
Producer of IP group traffic
eg. Camera/encoder
z2
74
Network After IGMP
c1
c1
Consumer of IP group
eg. Display/decoder
L2
SW
1
2
3
4
L3
1
RT
L2
SW
1
2
5
6
B
3
4
5
3
4
R
With pruning,
only the desired
recipients see
the traffic
A
6
L2
SW
2
1
2
3
4
z1
5
6
C
L2
SW
1
2
3
4
5
6
D
p
Producer of IP group traffic
eg. Camera/encoder
z2
75
GMRP: Generic Multicast Routing
• Same goal as IGMP – multicast pruning
• Layer 2 protocol i.e.can be done for payloads
other than TCP/IP
• IGMP predominates due to installed base
• Could find a following for real-time, industrial
protocols that are not over IP
• Shares GARP protocol with GVRP for
exchanging data between switches
76
Link Aggregation 802.1ad
• Aggregates several inter-switch links into a single
logical link
– Increases bandwidth incrementally
– Redundant connection – independent from STP
– Automatically shares load between links
• Traffic is distributed between links by ‘conversation’
using a simple algorithm involving the source and
destination addresses
• EtherChannel is Cisco’s proprietary equivalent
77
SNMP
• Simple Network Management Protocol
– Not that simple!
• Can get/set all switch parameters
• Traps very useful – event driven notification of
problems – can turn into emails, pages, etc. via
NMS (network management software)
• Uses MIBs (Management Information Database)
to define available data
• Many standard MIBs result in consistent data
API between vendors
78
RMON
• Remote Monitoring (RMON) is a standard monitoring
specification that enables various network monitors and
console systems to exchange network-monitoring data.
• RFC 2819 – RMON MIB
• RMON Monitoring Groups:
– Statistics: Contains statistics measured by the probe for each
monitored interface on this device.
– History: Records periodic statistical samples from a network
and stores them for later retrieval.
– Alarm: Periodically takes statistical samples from variables in
the probe and compares them with previously configured
thresholds. If the monitored variable crosses a threshold, an
event is generated.
– Event: Controls the generation and notification of events from
this device.
79
Networking Legacy Serial Devices
•
•
•
•
•
What to do about existing IEDs, RTUs, etc. that use serial
communications (e.g. RS232, RS422, RS485)?
Create an IP (UDP or TCP) “tunnel” through the Ethernet network
What comes in one end of the tunnel goes out the other end
Often called a “Serial Device Server”
Smart serial device servers also translate protocols
– eg. Modbus RTU -> Modbus TCP
•
Allows legacy devices to be accessed using Ethernet infrastructure
Serial IED
Ethernet SCADA
RS485
IP Network
Serial Device Server
Serial Device Server
Serial SCADA
80
RuggedSwitch™ & RuggedServer™
Product Overview
81
Rugged Operating System (ROS™)
¾ Zero Collisions: IEEE 802.3x Full Duplex Operation
¾ Priority Queuing: IEEE 802.1p for high priority real-time control
¾ VLAN: IEEE 802.1q for isolating real-time traffic
¾ Enhanced IEEE 802.1w Rapid Spanning Tree for fast fault
recovery
¾ IGMP Snooping for multicast filtering and management
¾ Network management: including SNMP, RMON, Port Mirroring
¾ Rich set of diagnostic tools
¾ Common firmware across all managed switches
¾ Simple firmware upgrade as new features become available
Fully Managed Switch Designed for Real-Time Control
82
RuggedSwitch™ RS900
Industrially Hardened – Managed Ethernet Switch
¾ up to 9 Ports
Optional Ports:
• Up to 3 Ports
• Fast 10/100BaseTx
• Fiber Optical (MMF/SMF)
• SC, ST, LC and MTRJ
• Bi-directional (single strand)
• Distances up to 90km
- Copper and Fiber combinations
¾ RuggedRated™ - Industrially Hardened
- EMI Immunity (IEC 61850-3, IEEE 1613),
- Operating Temperature (-40 to +85°C)
¾ ROS™ (Rugged Operating System)
- Advanced Layer 2 and 3 Network
Management
Fast Ethernet Ports:
• 6 - Fast Ethernet Ports
(10/100BaseTX)
¾ eRSTP™ for high speed (<5ms) network
fault recovery and redundancy
¾ Port rate limiting (128, 256, 512, 4000,
8000 kbps) for network traffic management
Integrated Power Supply
• Universal high-voltage range:
88-300VDC or 85 - 264VAC
• Popular low voltage DC ranges:
24VDC, 48VDC
¾ Hazardous Location Certification:
- Class 1 Division 2
Failsafe Output Relay
• Form-C contact output
• 1A@30VDC
83
RuggedSwitch™ RS900G
Industrially Hardened – Gigabit Managed Ethernet Switch
¾ Gigabit – Dual fiber optical 1000BaseX
ports allow for high-speed Gigabit
backbone (up to 70km).
Dual Gigabit Ports:
• Fiber Optical (MMF/SMF)
• Pluggable Optics (SFP)
• Bi-directional (single strand)
• Distances up to 70km
¾ RuggedRated™ - Industrially Hardened
- EMI Immunity (IEC 61850-3, IEEE 1613),
- Operating Temperature (-40 to +85°C)
¾ ROS™ (Rugged Operating System)
• 8 - Fast Ethernet Ports
(10/100BaseTX)
- Advanced Layer 2 and 3 Network
Management
¾ eRSTP™ for high speed (<5ms) network
fault recovery and redundancy
88-300VDC or 85 - 264VAC
24VDC, 48VDC
¾ Port rate limiting (128, 256, 512, 4000,
8000 kbps) for network traffic management
¾ Hazardous Location Certification:
- Class 1 Division 2
• 1A@30VDC
84
RuggedSwitch™ RSG2100
“Modularity plus Gigabit Ethernet”
¾ Modularity – 3 Gigabit Ports, 16 Fast Ethernet Ports (virtually any mix of fiber or copper desired)
¾ RuggedRated™ “Industrially Hardened”: IEC 61850-3, IEEE 1613, (-40 to +85°C)
¾ Integrated Dual Redundant Power Supplies 24Vdc, 48Vdc, or (88 – 300Vdc / 85 – 264Vac)
¾ Zero-Packet-Loss™ Technology for immunity to high levels of EMI
¾ ROS™ (Rugged Operating System) Advanced Layer 2 and 3 Management
¾ eRSTP™ for high speed (<5ms) network fault recovery and redundancy
¾ Port rate limiting (128, 256, 512, 4000, 8000 kbps) for network traffic management
85
Gigabit Ports:
• up to 3 Gigabit Ethernet Ports
• 10/100/1000 TX RJ45
• 1000SX Multimode
• 1000LX Singlemode
3 Gigabit Ports
88-300VDC or 85 - 264VAC
24VDC, 48VDC
• Dual Redundant (Optional)
• Parallel Load Sharing
• 1A@30VDC
Modularity (8x2)
Modular HMI:
• Front or Rear Mount
Mounting Options
• Panel/Din Rail
• 19” Rack Mount
• up to 16 Fast Ethernet Ports
• virtually any mix of fiber or copper desired
• 10/100TX RJ45
• 10FL Multi- and Singlemode
• 100FX Multi- and Singlemode
86
“9 Port Modular Managed Gigabit Ethernet Switch”
¾ Modularity – 9 Gigabit Ports (virtually any mix of fiber or copper desired)
¾ RuggedRated™ “Industrially Hardened”: IEC 61850-3, IEEE 1613, (-40 to +85°C)
¾ Zero-Packet-Loss™ Technology for immunity to high levels of EMI
¾ Integrated Dual Redundant Power Supplies 24Vdc, 48Vdc, or (88 – 300Vdc / 85 – 264Vac)
¾ ROS™ (Rugged Operating System) Advanced Layer 2 and 3 Management
¾ eRSTP™ for high speed (<5ms) network fault recovery and redundancy
¾ Port rate limiting (128, 256, 512, 4000, 8000 kbps) for network traffic management
87
Gigabit Ports:
• up to 3 Gigabit Ethernet Ports
• 10/100/1000 TX RJ45
• 1000SX Multimode
• 1000LX Singlemode
9 Gigabit Ports
88-300VDC or 85 - 264VAC
24VDC, 48VDC
• 1A@30VDC
Mounting Options
• Panel/Din Rail
Modular HMI:
Modularity:
• 5 available slots
• up to 9 ports
88
RSG2200/2100 Mounting
Front Mounting
HMI
Power
All communications ports out the front, power port from the rear
Rear Mounting
HMI
Power
Both power and communications ports from the rear, HMI port from front
89
RuggedSwitch™ RS969
“The World’s First IP65/IP67 Rated, Fully Managed, Industrial Ethernet
Switch with Gigabit”
¾ Waterproof: IP65 (Water Jets) and IP67 (Immersion)
¾ Industrial Operating Temperature: -40 to +85C
¾ High Immunity to EMI: Meets or exceeds IEC 61850-3, IEEE 1613, NEMA TS-2 and more ...
¾ Integrated Power Supplies: Low and high voltage ranges with true (N+1) redundancy option
¾ High Speed Fault Recovery: eRSTP™ delivers < 5ms per hop fault recovery performance
¾ Fully Managed: ROS™ delivers advanced networking and management features
¾ Gigabit: 2-Gigabit fiber optical waterproof ports for high-bandwidth applications
90
RS969 (M12 Connectors)
• 8 - Fast Ethernet Ports (10/100BaseTX)
• M12 Connectors
• High EMI immunity
• Transient and Surge protected
LED Indicators
• Link Activity per port
• Power and Alarm
Console Port:
• RS232 programming port
Water-proof Enclosure
• IP65 Rated (Water-Jet)
• IP67 Rated (Immersed)
• DIN Rail or Flush Mount
• Aluminum
• 1A@30VDC
Power Supply
88-300VDC or 85 - 264VAC
24VDC, 48VDC
• M12 Connector
Fiber Optical Gigabit Ethernet Ports
• 2 - Fiber Optical Gigabit Ethernet
Ports (1000BaseX)
• Fiber Optical (up to 25km)
• Waterproof covers when not in use
91
RS969 (RJ45 Connectors)
• 8 - Fast Ethernet Ports (10/100BaseTX)
• IP67 Rated RJ45 Connectors
• High EMI immunity
• Transient and Surge protected
LED Indicators
• Link Activity per port
• Power and Alarm
Console Port:
• RS232 programming port
Water-proof Enclosure
• IP65 Rated (Water-Jet)
• IP67 Rated (Immersed)
• DIN Rail or Flush Mount
• Aluminum
• 1A@30VDC
Fiber Optical Gigabit Ethernet Ports
• 2 - Fiber Optical Gigabit Ethernet
Ports (1000BaseX)
• Fiber Optical (up to 25km)
• Waterproof covers when not in use
Power Supply
88-300VDC or 85 - 264VAC
24VDC, 48VDC
• M23 Connector
• Dual-Redundant (option)
• Can be different sources!
92
RuggedServer™ RS400
Serial to Ethernet
“Industrially Hardened Serial Device
Server”
¾ Highly Integrated Device
¾ RuggedRated™ for Harsh Environments
- 4 isolated serial ports, a 4-port Managed Ethernet
Switch (fiber and copper options), V.90 Modem
¾ Multifunctional Operation
- Serial ports have 2kV of galvanic isolation to protect
against ground potential rise during ground faults
¾ Integrated Power Supplies
- serial-to-ethernet, remote access server, and router
functionality
¾ Managed Ethernet Switch
- Low and high voltage ranges
¾ Wide Operating Temperature Range
- -40 to +85C
- advanced networking features for fault-tolerant
networks suitable for real-time control
¾ High Immunity to EMI
- Meets or exceeds IEC 61850-3, IEEE 1613, NEMA TS-2
IEC 61000-6-2, IEC 61800-3
¾ Advanced Serial Functionality
- supports Modbus and DNP 3.0 protocols
- serial encapsulation; COM port redirection with serial IP
93
RuggedServer™ RS400
Serial to Ethernet
Serial Ports
• 4 - RS485/RS232 Ports
• 3kV Isolation per Port
Ethernet Ports
• 4-Port Ethernet Switch
• Fiber & Copper Ports
Mounting Options
• Panel/Din Rail
88-300VDC or 85 - 264VAC
24VDC, 48VDC
• (220 VDC / 250 VAC)
Integrated V.90 Modem
• 56 kbps
94
RuggedMC™ Media Converters
RMC: Ethernet Media Converter
RMC40: Ethernet Speed/Media Converter
Speed/Media Conversion:
• 10/100TX to 100FX (MMF/SMF)
• 10TÅÆ 100TX
• Dual 100FX Ports for optical Rings
or Port Redundancy
Media Conversion:
• 10T to 10FL (MMF/SMF**)
• 100TX to 100FX (MMF/SMF)
Field Hardened:
• IEC 61000-6-2, IEC 61800-3,
NEMA TS2, IEC61850, IEEE1613
• Integrated Power Supply: 24, 48 or
88 to 300Vdc / 85 to 264Vac
• -40 to 85°C Operating Temp.
RMC20: Serial Media Converter
4-Port Unmanaged
Switch
Field Hardened:
• IEC 61000-6-2, IEC 61800-3,
88 to 300Vdc / 85 to 264Vac
RMC30: Serial-to-Ethernet Media Converter
Media Conversion:
• RS232/485/422 to Fibre Optical
• RS232ÅÆRS485 Å ÆRS422
• Point-to-Point or “Optical Loop”
configurations supported
Field Hardened:
• IEC 61000-6-2, IEC 61800-3,
88 to 300Vdc / 85 to 264Vac
Media Conversion:
• RS232/485/422 to 10/100BaseTX
• 300bps - 230kbps serial speeds
2-Port Serial Device
Server
Field Hardened:
• IEC 61000-6-2, IEC 61800-3,
88 to 300Vdc / 85 to 264Vac
95
Industrial Power Supply
Power Supply 1
Power Supply 2
(Optional)
¾ Fully integrated power supply (no external adaptors)
¾ Dual redundant power supply option
¾ Universal high-voltage range:
88-300VDC or 85-264VAC
¾ Parallel load sharing with true N+1 redundancy
¾ Popular low voltage DC ranges:
24VDC, 48VDC
¾ Can be powered from different sources:
- e.g. PS 1 from 110VAC and PS 2 from 48VDC
- e.g. PS 1 from 125VDC and PS 2 from 220VAC
¾ CSA/UL 60950 safety approved to +85°C
- e.g. PS 1 24VDC and PS 2 from 125VDC
- Any combination!
96
Lunch break.
97
Overview of Layer 3 Networking
(Routers)
98
Router / Layer 3 Switch
•
Used to link a local network to a remote network
– Wide Area Network (WAN) such as Frame Relay
– Local Area Network (LAN) generally a Layer 3 Switch
– Metropolitan Area Network (MAN)
•
•
Works at the Network Layer (Layer 3)
Router which is connected to LAN is generally used as
default gateway for all devices on this network
• Routers make decisions on where to send data based
on source and destination IP addresses
• Routers can offer redundant paths when used with
multiple WAN interfaces
99
Networking Terminology
• LAN – Local Area Network
A local area network (LAN) is a computer network covering a local
area, like a Substation.
• MAN – Metropolitan Area Network
A Communications network that covers a geographical area such
as a County. MAN’s might be considered a LAN that spans a large
area such as multiple substations. Switches are generally used for
this type of Topology.
• WAN – Wide Area Network
A larger network, usually consisting of a collection of LANs that
spans a large geographical area. An example of a WAN would be
the Internet. Routers are generally used for this type of Topology
100
WAN Interfaces
• T1 1.544 MBPS
–
–
–
–
Channelized (24 X 64KBPS channels)
Unchannelized (Full T1 1.544 MBPS)
Digital
North America
• E1 2.016 Mbps
– Channelized (32 x 64kbps channels)
– Europe
• 56K DDS
– Analog
– Low speed (often used for Substation SCADA)
101
WAN Interfaces (cont)
• DSL
– ADSL – Asymmetric DSL (higher bandwidth
downstream than upstream I.E. 800 / 200)
– SDSL – Symmetric DSL (same bandwidth up/down)
• Modem / Low speed serial
– Used for Dial Backup or very low speed
communications
• Ethernet
– Used in MAN type design
– High speed
– Requires Fiber
102
WAN / MAN Interface Protocols
•
•
•
•
•
Point to Point Protocol (PPP)
Frame Relay
Dial Up (v.90 modem over PPP)
Ethernet
MPLS
103
Frame Relay
• Used to define PVCs (Private Virtual
Circuits)
A
Frame Relay Network
Access Links
C
B
C
104
Dial on Demand Routing
• Inexpensive backup for other WAN interfaces
PSTN
BRI/
PRI
A
Private
Network
X
BRI/
PRI
B
105
WAN Topology Choices
Point-to-Point
PPP, Cisco HDLC, ISDN
MP-to-Cloud
Frame Relay, ATM
Point-to-Cloud
Internet VPN
Internet
IPSec
MPLS-VPN
106
Router Function
•
Determine optimum routing paths through a network
• Lowest delay
• Highest reliability
•
Transport packets through the network
• Examines destination address in packet
• Makes a decision on which port to forward the packet
through
• Decision is based on the Routing Table
•
•
Interconnected Routers exchange routing tables in
order to maintain a clear picture of the network
In a large network, the routing table updates can
consume a lot of bandwidth
• a protocol for route updates is required (I.E. RIP, OSPF)
107
Basic IP Routing
• PC2 sends a packet to router B
• Router B matches the destination IP
address (162.11.5.1) to routing table
• Router B forwards packet out S0 to
Router A
162.11.5.0
1
A
S0
S1
162.11.5.1
Routing Table
Destination
Subnet
162.11.8.0
162.11.9.0
Next
Router
S0
S0
S1
162.11.6.0
S1
C
B
Router B
2
162.11.10.0
Outgoing
Interference
162.11.7.0
162.11.7.0
162.11.8.0
162.11.5.0
162.11.6.0
162.11.9.0
162.11.10.0
E0
S0
S0
S1
S0
S1
--------------162.11.8.1
-------162.11.8.1
162.11.6.3
108
Building the Routing Table
RIP Routing Update
162.11.10.0
162.11.5.0
162.11.9.0
162.11.5.0
1
A
S0
2
1
1
S1
162.11.5.1
Variety of IP Routing Protocols:
RIP
OSPF
EIGRP
IGRP
IS-IS
S1
Routing Table
Destination
Subnet
162.11.8.0
162.11.9.0
Outgoing
Interference
Next
Router
S0
S0
S1
162.11.6.0
S1
C
162.11.10.0
162.11.9.0
162.11.5.0
B
Router B
1
1
2
162.11.10.0
162.11.7.0
162.11.7.0
162.11.8.0
162.11.5.0
162.11.6.0
162.11.9.0
162.11.10.0
E0
S0
S0
S1
S0
S1
--------------162.11.8.1
-------162.11.8.1
162.11.6.3
RIP Routing Update
109
Routing Protocols Classes
Transit Network B
( Internet )
Network A
Network C
Exterior
Border Gateway Interior Gateway
Protocol
Protocol
(RIP, OSPF)
(BGP)
Interior Gateway
Protocol
(RIP, OSPF)
110
RIP v1 & v2
•
•
•
•
•
•
Routing Information Protocol
Very chatty (Consumes bandwidth)
Distance vector protocol
Useful for small subnets
Easy to install
Distributed
111
OSPF
• Common link-state protocol
• Decreases use areas by dividing each
area to smaller sub-areas and setting a
hierarchy.
• Uses Dijkstra’s Algorithm
• Centralized
• The preferred interior routing protocol on
the Internet.
112
BGP
•
•
•
•
Border Gateway Protocol
Path-vector
Scales well
Preferred Exterior Routing Protocol for
the Internet
• Complex to set up
113
VRRP
• Virtual Router Redundancy Protocol RFC 2338
• Provides layer 3 resiliency by allowing 2 or
more routers to act as a single “virtual router”.
– One becomes “Master”, others are “Backup”.
– “Clients” don’t need to know that they are talking to
a virtual router.
• Master uses special MAC address (VRMAC)
assigned from IANA -00-00-5E-00-01-{VRID}
– {VRID} is the VRRP Virtual Router Identifier
allowing up to 255 VRRP routers on a LAN.
114
Topology with VRRP
WAN
A
B
• WAN Interfaces can
be different
technologies (eg.
T1 and DSL)
Private
Network
115
Network Security Overview
116
Some History on Cyber
Security
¾ As stated in a 1990 report by the National Research
Council, "Tomorrow's terrorist may be able to do
more damage with a keyboard than with a bomb."
¾ The first widespread worm attack through
networked computers occurred in 1988 when
Robert Morris, Jr., a Cornell University student and
the son of a prominent NSA scientist, developed a
program that crippled approximately 6,200
computers and caused over $98 million in damage
in approximately 48 hours.
117
Security Statistics
Statistics from a variety of sources also support the notion that the cyber
threat faced by nations is growing in magnitude and consequence:
•
•
•
•
•
•
27 million Americans have suffered identity theft since 1999 (this includes
credit card, Social Security, and personal data). (Source: Federal Trade
Commission)
Over $222 billion in losses were sustained by the global economy as a result of
ID theft. (Source: Aberdeen Group June 2003 Report on the Economic Impact
of ID Theft)
4,700 Suspicious Activity Reports per Computer Intrusion were reported in
2003—a 100% increase. (Source: FINCEN, U.S. Treasury)
3600% increase in U.S. computer crime since 1997. FBI Director named Cybercrime the nation’s number one criminal problem. (Source: ITAA book "Long
Campaign”)
Between 1999 and 2003 in the U.S., attacks on computer servers increased by
over 530% to over 140,000 incidents for 2003.
The number of new vulnerabilities discovered in software is growing at 140%
per year and is now in excess of 4,000 per year. (Source: CERT/CC)
118
Potential Risk to Power Industry
•
An analysis of the North American electrical blackout of August 2003
provides more useful information: the CRS Report on Economic Impact of
Cyber Attacks indicated the power failure cost between $6 billion and $10
billion, disrupted production, affected earnings and profits, spoiled food
supplies, and increased first responder costs for some communities. Like a
cyber attack, there was little, if any, destruction of physical capital. (Source:
December 2004 Report of the Activities and Findings by the Chairman and
Ranking Member Subcommittee on Cybersecurity, Science, and Research &
Development of the U. S. House of Representatives Select Committee on
Homeland Security)
•
"Patrick H. Wood III, the chairman of the Federal Energy Regulatory Commission,
warned top electric company officials in a private meeting in January that they need
to focus more heavily on cyber security. Wood also has raised the issue at several
public appearances. Officials will not say whether new intelligence points to a
potential terrorist strike, but Wood stepped up his campaign after officials at the
Energy Department's Idaho National Laboratory showed him how a skilled hacker
could cause serious problems. . . . Describing his reaction to the demonstration,
Wood said: 'I wished I'd had a diaper on.'“ (reported by: Washington Post)
119
Network Security is Serious
Business
¾ 58 percent of companies surveyed reported authorized
users and employees as the source of a security breach
(1)
¾ Total annual cost of security breaches to corporations
$15B (2)
(1)
(2)
Source: DataMonitor PLC, New York
Source: PricewaterhouseCoopers survey
120
Vulnerabilities on the Rise
121
Network Vulnerabilities
•
Spoofing is pretending to be someone else in communications.
Spoofing is very simple in non-cryptographic protocols. IP spoofing
attack is manipulation of IP source address to present oneself as
the trusted party in the communication.
•
Denial of Service (DOS) includes extensive service requests to be
denied. Even just pinging a device extensively can disable the
appliance by keeping it busy with the service denial.
•
Replay Attacks are attacks when someone replays an old
message. An encrypted message cannot be read, but if the
moment is right the replay attack can be dangerous especially if it
is related to authentication.
122
Host Vulnerabilities
•
Virus - A virus is a manmade program or piece of code that
causes an unexpected, usually negative, event. Viruses are often
disguised games or images with clever marketing titles such as
“Viagra." Example: Netsky
•
Worm - Computer Worms are viruses that reside in the active
memory of a computer and duplicate themselves. They may send
copies of themselves to other computers, such as through email or
Internet Relay Chat (IRC). Example: Bagle, MyDoom
•
Trojan Horse - A Trojan horse program is a malicious program
that pretends to be a benign application; a Trojan horse program
purposefully does something the user does not expect. Trojans are
not viruses since they do not replicate, but Trojan horse programs
can be just as destructive. Example: Backdoor
123
Physical Vulnerabilities
• Network and host vulnerabilities can lead to
physical vulnerabilities:
–
–
–
–
Disabling network equipment and/or IEDs
Reconfiguring network equipment and/or IEDs
Gaining unauthorized access
Disabling access control and video surveillance
equipment
124
Network Zones
• Trusted – most secure section of the Network. Often called
the Inside.
• Untrusted – Least secure section of the Network. Generally
exposed to the public Internet
• DMZ (Demilitarized Zone) – A neutral zone which sits
between the trusted and untrusted zones of the network. The
DMZ is generally used to provide protection to Web and FTP
Servers.
• Perimeter – The border of a Network. Generally the
Boundary between the Untrusted section of a Network and the
Public Internet.
125
Security Strategy
Network Security Strategy of the Future
should include the following:
¾ Host based security
¾ Layered Security
- Firewall
- 802.1x Authentication / Authorization
- IDP
- Mac Filtering
- Email scanning
¾Strong Security Policy
- Antivirus
- VPN
- Password policy (key to success)
¾ Security Administration
- Email Policy
- Monitoring of Security Devices
-Strong configuration
¾ Physical Security
- Access Policy
- Use and Abuse Policy
- Program Change
126
Network Security NERC1300 and CIP
requirements
127
NERC Standards
CIP-002 to CIP-009 Standards as compared to sections
in Draft Standard 1300 – Draft 1
¾ NERC Cyber Security Goal:
“ensure that all entities responsible for the reliability
of the bulk electric systems of North America
identify and protect critical cyber assets that control
or could impact the reliability of the bulk electric
systems”
¾ Urgent action NERC 1200 cyber security
standard was initially adopted in August 2003
¾ Urgent action NERC 1200 renewed for a
second year in August 2004
¾ A permanent cyber security standard has
been under development and is expected to
be submitted to ballot later in 2006
(originally end of 2005)
¾ New standard was going to be called NERC
1300 but has changed to 8 separate
standards: CIP-002 to CIP-009
New Std #
Topic
Old Section
#
CIP-002-1
Critical Cyber Assets
1302
CIP-003-1
Security Management Controls
1301
CIP-004-1
Personnel and Training
1303
CIP-005-1
Electronic Security
1304
CIP-006-1
Physical Security
1305
CIP-007-1
Systems Security
Management
1306
CIP-008-1
Incident Reporting and
Response Planning
1307
CIP-009-1
Recovery Plans
1308
128
NERC Definitions
Critical Asset: Those facilities, systems, and equipment which, if destroyed, damaged, degraded, or
otherwise rendered unavailable, would have a significant impact on the ability to serve large quantities of
customers for an extended period of time, would have a detrimental impact on the reliability or operability
of the electric grid, or would cause significant risk to public health and safety.
Critical Cyber Assets: Those Cyber Assets essential to the reliable operation of Critical Assets.
Cyber Assets: Those programmable electronic devices and communication networks including hardware,
software, and data associated with bulk electric system assets.
Cyber Security Incident: Any malicious act or suspicious event that:
- Compromises, or was an attempt to compromise, the electronic or Physical Security Perimeter
of a Critical Cyber Asset, or,
- Disrupts or was an attempt to disrupt the operation of a Critical Cyber Asset.
Electronic Security Perimeter: The logical border surrounding the network or group of sub-networks (the
“secure network”) to which the Critical Cyber Assets are connected, and for which access is
Physical Security Perimeter: The physical border surrounding computer rooms, telecommunications
rooms, operations centers, and other locations in which Critical Cyber Assets are housed and for which
129
CIP-003 Security Management
Control
Identify and Document Policies and Procedures for
Security Management
¾ R5 – Access Control
- Logical and Physical Access by authorized personnel only
130
CIP-005 Electronic Security
Identify the Perimeter and How it Will be Defended
¾ R1 - Electronic Security Perimeter
- Identify the electronic security perimeter
- Identify access points
- All cyber assets inside the perimeter to be protected
- Critical assets are those that control/monitor the perimeter
- Specific case for dial-up access
¾ R2 – Electronic Access Control
- Enable only necessary ports and services
- Secure dial-up access
- Identify access controls and authentication methods
131
CIP-005 Electronic Security
¾ R3 – Monitoring Electronic Access Controls
- Log authorized access
- Detect unauthorized access attempts
- 24x7 monitoring
- Periodic review of logs
¾ R4 – Cyber Vulnerability Assessment
- Only necessary ports and services are enabled
- Discovery of modems
- Review of default accounts, default passwords, community strings
- Documentation of test results and action/remediation plan
¾ R5 – Documentation Review and Maintenance
- Annual review
132
CIP-007 System Security
Management
Monitor and Protect Critical Assets from Failure
¾ R3 – Ports and Services
- Only necessary ports and services are enabled
R6 – Account Management
- User accounts
- Remove or minimize shared accounts
- Audit trails
- Password management
¾ R7 – Security Status Monitoring
- Automated tools and processes to monitor
- Alerts for detected Cyber Security incidents
- Maintain logs to allow for root-cause analysis
133
CIP-009 Recovery Plans for
Critical Cyber Assets
Policies and Procedures for Business Continuity and
Disaster Recovery
¾ R4 – Backup and Restore
- Processes and procedures for backup and secure storage of information
134
RuggedCom NERC CIP
Compliance
Security Category
RuggedSwitch (ROS) and RuggedRouter
(ROX) Features
CIP Requirement
Passwords
User passwords
Multi-level passwords
CIP-003-1: R5.1.1
CIP-005-1: R2.1, R2.4
CIP-007-1: R6.1, R6.2.2, R6.3.1, R6.3.2
Network Management Security
SSH/SSL
Radius
SNMPv3
SNMP IP Restriction
CIP-003-1: R5.1.1
CIP-005-1: R2.1, R2.4
Network Security
Enable/Disable Ethernet Ports
Enable/Disable Services (Router)
MAC Based Port Security (ROS)
802.1x Port Security (ROS, Router - future)
802.1Q VLAN (ROS, Router - future)
CIP-005-1: R2.1, R2.1.1
CIP-007-1: R3
Routing Security
Firewall (Router)
VPN (Router)
IP Access Control
Intrusion Detection System (future)
CIP-003-1: R5.1.1
CIP-005-1: R2.1, R2.4
Logfiles, Traps, Alarms
System Logs
SNMP Traps
RMON
Alarms
Extensive Logging Capabilities (Router)
Remote Logging (Router)
CIP-005-1: R2.5, R3
CIP-007-1: R6.1, R6.1.3,
R7, R7.2, R7.3, R7.4
Configuration and Patch
Management
RS TFTP
RuggedVue
CIP-007-1: R4
CIP-009-1: R4
135
How Switches and Routers Provide
Security
136
Firewall
•
Helps to keep undesirable traffic out of your network
–
–
–
•
Examines source and destination address and blocks traffic
which does not meet predefined criteria
Block undesirable ports / services such as FTP, TFTP, File Sharing
First line of defense in protecting your network
–
–
–
–
Generally used on network perimeter
May protect multiple networks
Often provides Network Address Translation (NAT) to allow Trusted
users to access the Untrusted side of the network
•
Operates on Rules which are created by Network administrator
•
Other possible functions (select Firewall Manufacturers)
–
–
–
–
Virtual Private Network (VPN) terminating device
Intrusion Detection (IDS) or Intrusion Prevention (IDP) device
Antivirus Gateway (AV)
Email Scanner
137
Intrusion Detection (IDS) /
Prevention System (IPS)
•
Intrusion Detection / Prevention systems
analyze data in more detail than the
traditional Firewall
•
Uses a database of known signatures to take
action
– IDS will notify System Administrator
– IDP can be used to block suspect traffic
•
Can be integrated into Firewall solution
138
Virtual Private Network (VPN)
• VPN’s allow secure communication of data
between networks across the Internet or other
unsecured networks
•
Many different encryption schemes
• DES 56 bit encryption (Oldest least secure)
• 3DES 168 bit encryption (Most common today)
• AES up to 256 bit encryption Advanced Encryption
Standard
• (latest technology)
•
Can be used for RAS or Site to Site
communication
139
Management Security
¾ Password – Secures switch via password against
unauthorized configuration
¾ SSH / SSL – Extends capability of password protection
to add encryption of passwords and data as they cross
the network *
¾ Radius - Provides centralized password management
for Management passwords
140
Port Security
¾ Enable / Disable ports – Capability to disable ports so that traffic can
not pass
¾ 802.1Q VLAN – Provides the ability to logically segregate traffic
between predefined ports on switches
¾ Port based security – The ability to secure ports on a switch so only
specific Devices / MAC addresses can communicate via that port
¾ 802.1x – The ability to lock down ports on a switch so that only
authorized clients can communicate via this port. This generally
requires a user name / password be provided via software client on PC
or IED that communicates with the switch. A Radius server needs to
be present on the network to authenticate the user name and
password passed from the client to the switch.
2
Switch attempts to authenticate client
Client
3
1
ials
credent
wledges
r ack no
e
v
r
e
s
Radius
PC attempts to communicate with
network / Sends user name and
password
4
Secured Network
authenticated users are allowed access
141
RuggedRouter™ and RuggedVue™
Product Overview
142
RuggedRouter™ RX1000
“Industrially Hardened Cyber Security Appliance”
¾ Rugged Operating System on Linux (ROX™)
¾ Integrated Router/Firewall/VPN
¾ Wide Operating Temperature Range: -40 to +85C
¾ High Immunity to EMI: Meets or exceeds IEC 61850-3, IEEE 1613, NEMA TS-2 and more ...
¾ Integrated Power Supplies: Low and high voltage ranges with true (N+1) redundancy option
¾ RuggedRated™ for Harsh Environments
¾ Modular: Various Types and Configuration of Interface Ports
¾ 5 Year Warrantee
143
RX1000 Physical Features
Multiple Ethernet Ports:
• Quad 10/100 Mbps
• Fiber or Copper
• LC, ST, MTRJ, SC
V.90 Modem (Optional)
• 56 kbps
Multiple WAN ports
• Quad T1/E1
• Dual DSL
• Dual DDS 56/64kbps Activity
GPS/IRIG Ports
• Built-in GPS, Antenna Input
• Multiple IRIG-B Outputs
• Manchester, AM, Baseband,
IRIG-B Types
Modular HMI:
Mounting Options
• Panel/Din Rail
144
RX1000 Physical Features
88-300VDC or 85 - 264VAC
24VDC, 48VDC
• 1A@30VDC
Enclosure
• IP40
• 18 AWG Galvanized Steel
Operating Temperature
• -40C to +85C
• No Fans
EMI Immunity
• Meets IEEE 1613 (electric power substations)
• Exceeds IEC 61850-3 (electric utility substations)
• Exceeds IEEE 61800-3 (variable speed drive system)
• Exceeds IEC 61000-6-2 (generic industrial environment)
• Exceeds NEMA TS-2 (traffic control equipment)
145
RX1000 Key Features
Security Appliance Functions
¾ Integrated Router/Firewall/VPN
¾ Stateful Firewall with NAT
¾ Full IPSec Virtual Private Networking
¾ VPN with 3DES, DES, AES
¾ IDS (coming soon)
Protocols
¾ WAN: Frame Relay, PPP, PAP, CHAP Authentication, PPPoE (coming soon)
¾ IP: Routing, RIP/RIPII, OSPF, DHCP Agent
¾ Traffic shaping and policing
Management Tools
¾ Web Based GUI, SSH, CLI (command line interface)
¾ SNMP v2/v3
¾ Remote Syslog
¾ Rich set of diagnostics with logging and alarming
146
RuggedVue™ - Network
Management Software
Layer 3 IP View
Connectivity
Status
Hyperbolic
Tree
Device
Properties
147
RuggedVue™ - Network
Management Software
¾Graphic Visualization
“The exact network layout is represented with a hyperbolic tree
structure for ease of use, overview and tracking on a single page.”
¾Dynamic Discovery
“RuggedVue™ and its inherent dynamic nature will discover and
track all fixed devices in real time.”
¾Real Time Monitoring
“All devices and their connection, including wireless, in the
network are constantly monitored for their potential status.”
¾Documenting System
“RuggedVue™ allows linking to all types of documentation in
different web accessible locations.”
¾Client Server Architecture
“RuggedVue™ is a server based software that will deliver the
network data to all browsers on the network without any
specialized client software.
148
ROS™ and ROX™
Security Features
ROS™ = Rugged Operating System
ROX™ = Rugged Operating System On Linux
149
ROS™ Security Features
¾ Multilevel User Passwords
– Secures switch against unauthorized configuration
¾ SSH / SSL Encryption
– Encryption of passwords and data as they cross the network
¾ Enable / Disable ports
RuggedCom’s Embedded OS
for Networking Device
- Disable ports so that traffic can not pass
¾ 802.1Q VLAN (Virtual Local Area Network)
- Logically segregate traffic between predefined ports on switches
¾ MAC Based Port Security
- Secure ports so only specific Devices/MAC addresses can
communicate via that port
¾ 802.1x Port Based Network Access Control
- Lock ports to allow only authorized clients to communicate via the port
¾ Radius
- Centralized password management
¾ SNMPv3
- Encrypted authentication and access security
150
ROS™ Multilevel User Passwords
¾ Three password levels:
Admin:
Ability to change configuration, execute commands and view data.
Operator: Ability to execute commands and view data.
Guest:
Ability to view data only.
¾ Passwords can be up to 15 alphanumeric or special characters
¾ Radius back end can be used for centralized password management
151
ROS™ SSL Web Server Security
¾ Secure Sockets Layer (SSL) encrypts all http traffic from
client to and from server
¾ Prevents snooping of management session and
harvesting of passwords
¾ Supports by all commercial Web browsers
¾ SSL MD5 and DES
152
ROS™ SSH Command Line Security
¾ Secure Shell (SSH) provides a secure alternative to Telnet
¾ sftp for secure file transfers
¾ slogin for secure CLI automation
¾ MD5 authentication and DES encryption
¾ PuTTY is a free SSH client
153
ROS™ Physical Port Security
¾ ROS™ allows disabling of unused physical Ethernet
ports
¾ Disabling a port stops all traffic flow
¾ You can’t be any more secure that that!
154
ROS™ Port Security & 802.1x
¾ MAC based port security can restrict access to a single Ethernet device
- Innovative auto-learn feature makes it easy to use MAC security
¾ 802.1x port security uses standards based authentication mechanism with
Radius server backend
¾ Any unauthorized access attempt generates an alarm and SNMP traps; portt
can be subsequently locked out to prevent repeated access attempts
155
ROS™ VLAN Security
¾ 802.1Q based VLANs restrict the broadcast domain of a network
¾ Can be used to isolate hyper-secure network devices from general traffic
¾ VLAN is equivalent to a physically separate network from a security standpoint
¾ A router is required to bridge between VLANs
¾ ROS™ supports both port based VLANs and GVRP
156
ROS™ SNMP v3
¾ SNMP v3 provides secure management of ROS™
¾ ROS™ uses standards based MIBs to ensure compatibility with
wide variety of SNMP based NMS software packages
¾ Users are classified into groups; different groups can have different
access to MIB information
¾ Backwards compatible with v1 and v2c SNMP users
157
ROX™ Firewall Security
¾ Based on Linux Netfilter, Iptables, and Shorewall
¾ Stateful IP firewall, IP Masquerading(NAT), Port
Forwarding(DMZ),
158
ROX™ Firewall Rules
¾ Interfaces grouped into zones
¾ Default policies determine overall behavior of zones
¾ Rules define behavior for specific source IP address
and port number
159
ROX™ IPSec VPN
¾ IPSec allows secure tunnels through untrusted networks
¾ Supports network-network or network-host VPNs
¾ Allows either pre-shared or public key cryptography
¾ DES, 3DES, AES encryption protocols supported
160
??? Questions ???
161
Thank you.
Angelo Rizzo – International Sales Director
Roger Moore – Vice President Engineering
162

nsa 100sx

Transcription

Similar documents

RuggedRouter™ RX1000 - Signal Control Products, Inc.

FieldSmart FxDS Tie Panels

FRM220A Ethernet Aggregation Solution

RX1000/RX1100 Installation Guide

Cubee 615 - Maquipesa

AX-3000 - Uniwell EPOS systems

RoomWizard® II Scheduling System Technical Specifications

GIGABIT ABIT

csi global connect-internet remote access

High Performance, High Availability Ethernet Access for