docLDC-Track _2013_GTA_Blackout_Story
download 
Report Transcription
LDC-Track _2013_GTA_Blackout_Story
Areas Affected Weather Radar July 8, 2013 at 1700 EDT (2100 GMT) 126 millimetres of rain fall was recorded over a span of several hours, causing flash flooding throughout Toronto 1400 passengers were trapped on a waterlogged GO Train for up to seven hours Areas Affected Customers Affected 500,000 customers associated with 6 local distribution companies and 4 major industrial customers Hydro One Equipment Affected • (25) 230 kV circuits tripped • (7) 115 kV circuits tripped • 30 transformer stations interrupted • Multitude of communications and information technology equipment Transmission Supply to Toronto Area PARKWAY TS CAVANAGH TS IBM TS FAIRCHILD TS LESLIE TS RICHVIEW TS MARKHAM MTS #1 HURONTARIO SS PORTLANDS PORTLANDS ENERGY ENERGY CENTRECENTRE CHURCHILL MEADOWS TS COOKSVILLE TS CHURCHILL MEADOWS TS MANBY TS Generating Station GTA Load Supply Prior to the Event PARKWAY TS CAVANAGH TS IBM TS FAIRCHILD TS LESLIE TS MARKHAM MTS #1 HURONTARIO SS PORTLANDS PORTLANDS ENERGY CENTRECENTRE ENERGY Supply from Leaside TS CHURCHILL MEADOWS TS Supply from Manby TS Generating Station Sequence of GTA Load Loss on July 8 PARKWAY TS CAVANAGH TS IBM TS FAIRCHILD TS LESLIE TS MARKHAM MTS #1 W E DESN HURONTARIO SS YARROW TS PORTLANDS ENERGY CENTRE W E COOKSVILLE TS CHURCHILL MEADOWS TS OAKVILLE TS FORD OAKVILLE TS Pre-Event System Demand: 24,000 MW Sequence of GTA Load Loss on July 8 PARKWAY TS CAVANAGH TS IBM TS FAIRCHILD TS LESLIE TS MARKHAM MTS #1 W E DESN HURONTARIO SS YARROW TS PORTLANDS ENERGY CENTRE W E COOKSVILLE TS Time: 4:59PM CHURCHILL MEADOWS TS OAKVILLE TS FORD OAKVILLE TS Load Lost: 116 MW Sequence of GTA Load Loss on July 8 PARKWAY TS CAVANAGH TS IBM TS FAIRCHILD TS LESLIE TS MARKHAM MTS #1 W E DESN HURONTARIO SS YARROW TS PORTLANDS ENERGY CENTRE W E COOKSVILLE TS Time: 5:00PM CHURCHILL MEADOWS TS OAKVILLE TS FORD OAKVILLE TS Load Lost: 453 MW Sequence of GTA Load Loss on July 8 PARKWAY TS CAVANAGH TS IBM TS FAIRCHILD TS LESLIE TS MARKHAM MTS #1 W E DESN HURONTARIO SS YARROW TS PORTLANDS ENERGY CENTRE W E COOKSVILLE TS Time: 5:03PM CHURCHILL MEADOWS TS OAKVILLE TS FORD OAKVILLE TS Load Lost: 714 MW Sequence of GTA Load Loss on July 8 PARKWAY TS CAVANAGH TS IBM TS FAIRCHILD TS LESLIE TS MARKHAM MTS #1 W E DESN HURONTARIO SS YARROW TS PORTLANDS ENERGY CENTRE W E COOKSVILLE TS Time: 5:05PM CHURCHILL MEADOWS TS OAKVILLE TS FORD OAKVILLE TS Load Lost: 1262 MW Sequence of GTA Load Loss on July 8 PARKWAY TS CAVANAGH TS IBM TS FAIRCHILD TS LESLIE TS MARKHAM MTS #1 W E DESN HURONTARIO SS YARROW TS PORTLANDS ENERGY CENTRE W W E COOKSVILLE TS Time: 5:12PM CHURCHILL MEADOWS TS OAKVILLE TS FORD OAKVILLE TS Load Lost: 2366 MW Sequence of GTA Load Loss on July 8 PARKWAY TS CAVANAGH TS IBM TS FAIRCHILD TS LESLIE TS MARKHAM MTS #1 W E DESN DESN HURONTARIO SS YARROW TS PORTLANDS ENERGY CENTRE W W E COOKSVILLE TS Time: 5:26PM CHURCHILL MEADOWS TS OAKVILLE TS FORD OAKVILLE TS Load Lost: 3302 MW Sequence of GTA Load Loss on July 8 PARKWAY TS CAVANAGH TS IBM TS FAIRCHILD TS LESLIE TS MARKHAM MTS #1 W E DESN DESN HURONTARIO SS YARROW TS PORTLANDS ENERGY CENTRE W W E COOKSVILLE TS Time: 6:26PM CHURCHILL MEADOWS TS OAKVILLE TS FORD OAKVILLE TS Load Lost: 3398 MW What Was Flooded? At Richview • 230 kV West relay building • 230 kV East relay building • Telecom building • Back-Up Control Centre • Integrated Telecommunications Management Centre Storm water flooded the yard cable trenches and entered the relay buildings via cable penetrations in the basement walls Storm water also entered the Telecom Building via cable penetrations, sump pit overflow/backflow due to the loss of AC power to the sump pumps. Richview TS Relay Buildings Relay building basements had 6 ft of water, enough to reach the ceiling cable trays. 2-3 ft of water in the Telecom building basement Richview TS Relay Buildings Relay building basements had 6 ft of water (enough to reach the ceiling cable trays) Terminal racks for protection and control equipment (including breaker trip and control circuit) connections were submerged, resulting in various short circuits. Richview TS Control/Telecom/ITMC Building When AC grid power was lost, a backup generator supplied the Telecom building IT assets The backup generator tripped due to water intake, impacting Critical IT equipment (telecom, power system, and business) What Was Flooded? At Manby • Control room building • 230 kV relay building • 115 kV East relay building • 115 kV West relay building Stormwater flooded the yard cable trenches and entered the relay/control room buildings via cable penetrations in the basement walls Manby TS Control building had 5 ft of water in the sub basement. Lower tier of the station service (250 V) batteries were submerged. Relay buildings had up to 3 ft of water in their basements Manby TS The DC station service caused galvanic corrosion to circuitry that was submerged within the relay and control building basements Manby TS The Manby hubsite was lost including SCADA control and telemetry for the following stations: Manby TS (All yards) Dufferin Strachan Wiltshire Fairbank Horner Runnymede John Sequence of GTA Load Restoration PARKWAY TS CAVANAGH TS IBM TS FAIRCHILD TS LESLIE TS MARKHAM MTS #1 W E DESN RICHVIEW TS HURONTARIO SS WILTSHIRE TS RUNNYMEDE TS YARROW TS CECIL TS DUFFERIN TS W PORTLANDS ENERGY CENTRE E COOKSVILLE TS CHURCHILL MEADOWS TS OAKVILLE TS FORD OAKVILLE TS Load to Restore: 3398 MW Sequence of GTA Load Restoration PARKWAY TS CAVANAGH TS IBM TS FAIRCHILD TS LESLIE TS MARKHAM MTS #1 W E DESN RICHVIEW TS HURONTARIO SS WILTSHIRE TS RUNNYMEDE TS YARROW TS CECIL TS DUFFERIN TS W PORTLANDS ENERGY CENTRE E COOKSVILLE TS CHURCHILL MEADOWS TS Time: July 8 @ 6-7PM OAKVILLE TS FORD OAKVILLE TS Load to Restore: 3061 MW Sequence of GTA Load Restoration PARKWAY TS CAVANAGH TS IBM TS FAIRCHILD TS LESLIE TS MARKHAM MTS #1 W E DESN RICHVIEW TS HURONTARIO SS WILTSHIRE TS RUNNYMEDE TS YARROW TS CECIL TS DUFFERIN TS W PORTLANDS ENERGY CENTRE E COOKSVILLE TS CHURCHILL MEADOWS TS Time: July 8 @ 7-8PM OAKVILLE TS FORD OAKVILLE TS Load to Restore: 3006 MW Sequence of GTA Load Restoration PARKWAY TS CAVANAGH TS IBM TS FAIRCHILD TS LESLIE TS MARKHAM MTS #1 W E DESN RICHVIEW TS HURONTARIO SS WILTSHIRE TS RUNNYMEDE TS YARROW TS CECIL TS DUFFERIN TS W PORTLANDS ENERGY CENTRE E COOKSVILLE TS CHURCHILL MEADOWS TS Time: July 8 @ 8-9PM OAKVILLE TS FORD OAKVILLE TS Load to Restore: 2954 MW Sequence of GTA Load Restoration PARKWAY TS CAVANAGH TS IBM TS FAIRCHILD TS LESLIE TS MARKHAM MTS #1 W E DESN RICHVIEW TS HURONTARIO SS WILTSHIRE TS RUNNYMEDE TS YARROW TS CECIL TS DUFFERIN TS W PORTLANDS ENERGY CENTRE E COOKSVILLE TS CHURCHILL MEADOWS TS Time: July 8 @ 9-10PM OAKVILLE TS FORD OAKVILLE TS Load to Restore: 1851 MW Sequence of GTA Load Restoration PARKWAY TS CAVANAGH TS IBM TS FAIRCHILD TS LESLIE TS MARKHAM MTS #1 W E DESN RICHVIEW TS HURONTARIO SS WILTSHIRE TS RUNNYMEDE TS YARROW TS CECIL TS DUFFERIN TS W PORTLANDS ENERGY CENTRE E COOKSVILLE TS CHURCHILL MEADOWS TS Time: July 8 @ 10-11PM OAKVILLE TS FORD OAKVILLE TS Load to Restore: 1296 MW Sequence of GTA Load Restoration PARKWAY TS CAVANAGH TS IBM TS FAIRCHILD TS LESLIE TS MARKHAM MTS #1 W E DESN RICHVIEW TS HURONTARIO SS WILTSHIRE TS RUNNYMEDE TS YARROW TS CECIL TS DUFFERIN TS W PORTLANDS ENERGY CENTRE E COOKSVILLE TS CHURCHILL MEADOWS TS Time: July 8 @ 11-Midnight OAKVILLE TS FORD OAKVILLE TS Load to Restore: 820 MW Sequence of GTA Load Restoration PARKWAY TS CAVANAGH TS IBM TS FAIRCHILD TS LESLIE TS MARKHAM MTS #1 W E DESN RICHVIEW TS HURONTARIO SS WILTSHIRE TS RUNNYMEDE TS YARROW TS CECIL TS DUFFERIN TS W PORTLANDS ENERGY CENTRE E COOKSVILLE TS CHURCHILL MEADOWS TS Time: July 9 @ Midnight-1AM OAKVILLE TS FORD OAKVILLE TS Load to Restore: 629 MW Sequence of GTA Load Restoration PARKWAY TS CAVANAGH TS IBM TS FAIRCHILD TS LESLIE TS MARKHAM MTS #1 W E DESN RICHVIEW TS HURONTARIO SS WILTSHIRE TS RUNNYMEDE TS YARROW TS CECIL TS DUFFERIN TS W PORTLANDS ENERGY CENTRE E COOKSVILLE TS CHURCHILL MEADOWS TS Time: July 9 @ 2-3AM OAKVILLE TS FORD OAKVILLE TS Load to Restore: 308 MW Sequence of GTA Load Restoration PARKWAY TS CAVANAGH TS IBM TS FAIRCHILD TS LESLIE TS MARKHAM MTS #1 W DESN RICHVIEW TS HURONTARIO SS WILTSHIRE TS RUNNYMEDE TS YARROW TS CECIL TS DUFFERIN TS W PORTLANDS ENERGY CENTRE E COOKSVILLE TS CHURCHILL MEADOWS TS Time: July 9 @ 5:19AM OAKVILLE TS FORD OAKVILLE TS Load to Restore: 222 MW Sequence of GTA Load Restoration PARKWAY TS CAVANAGH TS IBM TS FAIRCHILD TS LESLIE TS MARKHAM MTS #1 W DESN RICHVIEW TS HURONTARIO SS WILTSHIRE TS RUNNYMEDE TS YARROW TS CECIL TS DUFFERIN TS W PORTLANDS ENERGY CENTRE E COOKSVILLE TS CHURCHILL MEADOWS TS OAKVILLE TS FORD OAKVILLE TS July 9th 7-8AM Voltage Reduction Begins July 9th 10-11AM Load Shedding Begins represents overloading Load Shed up to 86 MW Sequence of GTA Load Restoration PARKWAY TS CAVANAGH TS IBM TS FAIRCHILD TS LESLIE TS MARKHAM MTS #1 DESN RICHVIEW TS HURONTARIO SS WILTSHIRE TS RUNNYMEDE TS YARROW TS CECIL TS DUFFERIN TS W PORTLANDS ENERGY CENTRE E COOKSVILLE TS CHURCHILL MEADOWS TS Time: July 9 @ 5:18PM OAKVILLE TS FORD OAKVILLE TS represents overloading Load to Restore: 189 MW Load Shed up to 86 MW Sequence of GTA Load Restoration PARKWAY TS CAVANAGH TS IBM TS FAIRCHILD TS LESLIE TS MARKHAM MTS #1 DESN RICHVIEW TS HURONTARIO SS WILTSHIRE TS RUNNYMEDE TS YARROW TS CECIL TS DUFFERIN TS W PORTLANDS ENERGY CENTRE E COOKSVILLE TS CHURCHILL MEADOWS TS OAKVILLE TS FORD OAKVILLE TS July 9th 7-8PM Load Shedding Ends July 9th 8-9PM Voltage Reduction Ends on Cecil x Esplanade Load to Restore: 189 MW Sequence of GTA Load Restoration PARKWAY TS CAVANAGH TS IBM TS FAIRCHILD TS LESLIE TS MARKHAM MTS #1 DESN RICHVIEW TS HURONTARIO SS WILTSHIRE TS RUNNYMEDE TS YARROW TS CECIL TS DUFFERIN TS PORTLANDS ENERGY CENTRE W COOKSVILLE TS CHURCHILL MEADOWS TS Time: July 9 @ 10:18PM OAKVILLE TS FORD OAKVILLE TS Load to Restore: 141 MW Sequence of GTA Load Restoration PARKWAY TS CAVANAGH TS IBM TS FAIRCHILD TS LESLIE TS MARKHAM MTS #1 DESN RICHVIEW TS HURONTARIO SS WILTSHIRE TS RUNNYMEDE TS YARROW TS CECIL TS DUFFERIN TS PORTLANDS ENERGY CENTRE W COOKSVILLE TS CHURCHILL MEADOWS TS OAKVILLE TS July 10th 1PM Voltage Reduction Ends on Leaside x Wiltshire x Manby East FORD OAKVILLE TS Load to Restore: 141 MW Sequence of GTA Load Restoration PARKWAY TS CAVANAGH TS IBM TS FAIRCHILD TS LESLIE TS MARKHAM MTS #1 DESN RICHVIEW TS HURONTARIO SS WILTSHIRE TS RUNNYMEDE TS YARROW TS CECIL TS DUFFERIN TS PORTLANDS ENERGY CENTRE COOKSVILLE TS CHURCHILL MEADOWS TS Time: July 10 @ 2:24PM OAKVILLE TS FORD OAKVILLE TS Load to Restore: 0 MW What Went Well (1) • Good information exchange occurred between the Ontario Grid Control Centre (OGCC) and the Independent Electricity System Operator (IESO). Both entities attempted to share plans in advance to allow time for adequate system assessments and studies, with the intent to mitigate delays to restoration and the declaration of equipment available for service. • Good communication and information exchange occurred between the OGCC and affected entities, specifically with Toronto Hydro Electric Systems Limited (THESL) as they were largely impacted. The OGCC continually kept THESL abreast of restoration and network reinforcement plans, and coordinated switching mitigated delays. • OGCC made a conscious decision to restore circuits with fully functional breaker failure protection primarily for personnel safety, further mitigating the possibility of uncleared faults while staff worked in the yards. This also was a benefit for enforcing network reliability. What Went Well (2) • The restoration plan also took into consideration prioritized power restoration to both the IESO and THESL control rooms as they were running on diesel generators; ensuring their continued functionality. Also, there were accommodations for THESL priority loads e.g. water pumping station. • Staff worked safely with no reported injuries/incidents while responding to the emergency and during the restoration effort. • Corporate Relations provided ongoing updates to the Ministry of Energy, City of Toronto’s Emergency Operations Centre (EOC) with the understanding that they were communicating with key city staff and elected officials as well as THESL. In addition, Hydro One Networks Inc. (HONI) continued to issue press releases, tweet and proactively make calls to media when new information was available Major Challenges (1) • Inspection, assessment and reverification of all equipment that was affected by water damage, and in some cases replacement and installation of new equipment. • The Network Management System (NMS) state estimator had to compensate for the massive loss of telemetry which caused suspect analysis e.g. thermal monitoring of assets. Field confirmed telemetered values were manually inputted by the OGCC Controllers to increase the accuracy of the analysis. • Impairment of computer and networking systems affected the OGCC’s ability to operate, monitor and assess the power system. Also, some Local Distribution Company’s (LDC) operability at stations was impacted because their Inter Control Centre Protocol (ICCP) link with HONI is routed through Richview SONET ring Major Challenges (2) • All communications, telecom, control and telemetry were lost from Richview due to flooding and consequent power loss. Control and telemetry were also impacted for multiple other stations in the GTA. This required Protection and Control (P&C) and Electrical Maintenance Department (EMD) personnel dispatched to stations to perform switching via the station mimic board or the Local Control Computer (LCC). • Loss of AC station service which impaired many key station functions, such as station lights and power to buildings. This required crews to perform work in non-ideal situations; however they carried out remedial measures in a safety conscience manner. • Mechanical or pneumatic issues with equipment that required correction before it could be operated thus delaying switching sequences. “Key” Recommendations (1) The following recommendations were identified as corrective actions to address causal factors and mitigate the risk of exposure in future scenarios. Review and evaluate redundancy requirements for key computer and telecommunication services integral to OGCC’s operability of the power system. A review should be undertaken to assess and determine corrective methods for vulnerabilities in the security system. In addition, procedures should be reviewed and/or developed to mitigate the risk of exposure in future scenarios. Review and evaluate the setup of the magnetic locks and access to cyber secure locations, with specific attention given to situations where locks fail close upon loss of power thereby sealing the room. “Key” Recommendations (2) Explore opportunities for IESO pre-approved load restoration plans during localized emergencies. HONI should engage police services to explore a methodology to facilitate staff and/or equipment transfers to required HONI sites during emergencies. “Key” Recommendations (3) Seal watertight all cable penetrations and window wells at Richview TS and Manby TS. Re-design the diesel backup supply system for the Richview TS Telecom Building sump pumps to ensure it automatically starts-up. Identify all transmission station buildings with critical power system equipment located in basements and, where appropriate, develop a mitigation plan for incorporation into the 5 year investment plan. Determine if changes to the Hydro One design criteria for transmission station site drainage systems are required (i.e. design for a 200 year one hour storm event). “Key” Recommendations (4) Given the magnitude of the flood and its widespread impact on Richview and Manby equipment, the possibility of accelerating work programs (scheduled maintenance) should be reviewed to ensure full operability of all affected assets. Due to the evacuation of Integrated Telecommunications Management Centre (ITMC) personnel to their backup site, the OGCC was unable to obtain an evaluation of the impacts to the underlying network which supports control and telecommunication. Review and evaluate the availability of redundant telecommunication networks for the OGCC during an emergency. “Key” Recommendations (5) HONI has revised its crisis protocol - The VP of Corporate Relations will call the VP of Corporate Communications at Toronto Hydro to set context and develop joint messaging as soon as an incident occurs. HONI has conducted a post-event debrief with the City of Toronto’s Emergency Operations Centre (EOC), and they have been directed to contact HONI’s media and elected official line for emergency information. The line is staffed on a 24-7 basis by members of HONI’s corporate communications media team. The nature and extent of the flooding damage was unprecedented and reconfiguring the local power system had never been undertaken in this manner before. It is recommended that HONI continue to provide estimated times of restoration only where possible. The End Hydro One Chief Operating Officer Peter Gregg (left) speaks at a news conference with Ontario Energy Minister Bob Chiarelli at the Manby Transformer Station Wednesday July 17, 2013. Manby was one of the key stations knocked out during a massive July 8, 2013 flood in Toronto. http://www.cp24.com/news/energy-minister-praises-response-following-flood1.1370811
