Skip booklet index and go to page content

The Toronto 2015 Pan and Parapan American Games Experience

7. Integration Tests and Contingency Plans

Leading up to the Games, a significant amount of testing was required to ascertain that operational software and hardware were installed and fully functional and that the Mesonet data was available to forecasters for their analysis and interpretation. Forecasters and briefers needed to be trained on the services required of them during the Games. Their training sessions would also include information on new hardware and software, atmospheric monitoring Mesonet datasets and high-resolution numerical weather prediction and air quality model products that would be available to support the Games alerting program. The final lead-up year marked a substantial number of innovations and documented procedures to allow the teams to have both equipment and staff in place and fully operational to meet expectations. This included contingency planning to determine and document alternate courses of action for the provision of services in the event of an outage of the OSPC or either of the two briefing units.

7.1 Intensive Operating Period and Integration Testing

The concept of an Intensive Operating Period (IOP) was to run the “forecast, alerting and monitoring systems” from end to end just prior to and during the Games with integrated testing periods scheduled and planned as part of the process leading up to the Games. The testing would verify and validate that all instruments, workstations, forecasting work processes and staff were ready for operations. Although there was always a requirement to perform end-to-end testing and to test a new contingency plan, the IOP idea was conceived by the Chair of the Steering Committee in the fall of 2014, who asked that intensive testing be conducted over a period of time in preparation for the Games. The plan was endorsed by all of the Project’s Working Groups, and by October 2014, by the Steering Committee as well. What was special about the IOP was that the scope of the testing extended beyond the operational components of the forecasting services to include the offsite briefing offices. The IOP component was not part of the original Treasury Board Submission but became an integral part of ECCC’s implementation plan.

When the IOP test planning was being integrated into the overall implementation plan, it became obvious that all testing could not begin immediately as conceived, as there were critical IT systems upgrades that were required to have our Integrated Workstations operationally ready for the Games. There were also monitoring instruments and equipment that needed to be brought online. Thus, a staged approach was initiated so that over time, from the fall of 2014 to the start of the Pan Am Torch Relay in May 2015, testing and certification would be a phased part of our work and implementation. These phases were called “Integration Tests” and were the lead up to the start of the actual Intensive Operations that began one week before the start of the Pan Am Games.

7.1.1 Integration Details

The purpose of the Integration Tests was to conduct end-to-end testing of operations (i.e., testing of the various components of the complex systems, identifying any gaps and correcting them). In the lead up to final readiness and operations during the Games, five Integration Tests took place approximately one month apart from November 2014 to April 2015. Integration Tests validated the following components of the IOP: People/Training/Contingency; Monitoring/Data Flow; Tools; Prediction/Production; Dissemination; Science; and Project Management Activities where appropriate. The integrations were successive and progressively elaborated with increasing scope, and each primarily focused on one or two major themes.

The information below explains in some detail what was covered during each integration testing period. The integration steps have been colour coded to match the diagram that follows showing the correlation and progression of the Integration Tests.

  • Integration 1 – Testing focused on the basic hardware and software functionality of the ISW, observational data being delivered to this system including data from the Mesonet, as well as the capability to issue venue forecasts and alerts.
  • Integration 2 – The testing focus was similar to the scope of Integration 1, but included testing of two ISWs installed in the OSPC, the A-B switch (i.e., the “switch” allowed a forecaster to change from the ISW to the IFW) and forecast production software.
  • Integration 3 – Testing continued on the ISW in the OSPC. In addition, initial testing of an ISW in the Winnipeg Storm Prediction Centre Contingency Office, and the IPAW.
  • Integration 4 – Testing continued on the ISW in the OSPC and the IPAW and briefer/forecaster tools outside of the ECCC network. New testing focused on venue alerts being transmitted to EC Alert Me (see Section 9.6) and their availability to the briefers, the IRW and ECPASS (see Section 11.10).
  • Integration 5 – Testing simulated, as much as possible, end-to-end operations (i.e., as it would be during the Games). In addition, it included user acceptance testing before final operational readiness, contingency and training.

Figure 13. ECCC’s Intensive Operating Period Plan

A graphic (see long description below)

Description

A color-coded graphic showing the:

  • correlation and progression of the five integration steps in Environment and Climate Change Canada’s Intensive Operating Period Plan that were approximately one month apart from November 2014 through May 2015
  • Final Operational Readiness, Contingency and Training Period in May and June 2015
  • Intensive Operating Period in July and August 2015.

The steps are correlated with the different components of the IOP, where appropriate:

  1. People/Training/Contingency;
  2. Monitoring/Data Flow;
  3. Tools;
  4. Prediction/Production;
  5. Dissemination;
  6. Science; and
  7. Project Management Activities.

The critical path involves the first five
components (ab, b, c, d, e). The dates of each of the five integration steps are displayed in individual boxes on the graphic’s timeline, as well as the relevant components as defined above:

  • Integration 1: Nov 3–14, 2014 (b, c, d, g)
  • Integration 2: Dec 8–19, 2014 (b, c, d, g)
  • Integration 3: Jan 26–Feb 6, 2015 (a, b, c, d, e, g)
  • Integration 4: Mar 9–12, 2015 (b, c, e, f, g)
  • Integration 5: Apr 7–17, 2015 (a, b, c, d, e, f, g) (Ninjo 1.8.3 implemented)

7.1.2 Results

Documentation for each integration testing period included: a detailed plan; verification of test setup checklist; testing matrix; test cases; and a summary report. Gaps/Risks were tracked from each integration testing period and addressed during subsequent Integration Tests.

Based on the scope of the five integrations and their results, the Integration Tests were deemed successful. In summary, integration testing involved validating data flow to the ISW and IPAW (inside the ECCC network) on a staging environment inside the ECCC network. The ISW, IPAW and IRW hardware and software were proven to be robust. The processes for issuing and disseminating venue forecast and alerts were successful. The integration testing ultimately prepared Operations for the Final Operational Readiness, Contingency & Training Period (May 1–July 5) and Intensive Operations (July 6–August 15).

7.2 Training

Training was a critical preparatory step towards Games operations. There was a need to identify the differences between what was considered to be standard operating procedures and generation of products and what was required for the purposes of the Games. New processes, such as the issuance of venue-specific alerts and forecasts within existing public forecast areas, required a review of product distribution applications. Trainers mitigated another potential risk by travelling to the contingency offices to provide the same training to forecasters who could be handed operational reins should any emergency situation arise requiring the shutdown of the OSPC (see Section 7.3). A new level of close coordination between forecast and briefing teams was also necessary. The teams were trained in the use of Mesonet observations, new display applications arising from this Mesonet and high-resolution experimental numerical weather products. Innovative technologies were deployed, including Doppler LiDAR instruments, mobile car top weather stations and a sensor array for a three-dimensional (3D) total lightning solution. These all required training for interpretation. Briefing teams located at the MOC and the UCC were tasked with training and/or interpreting these data and products for outside clients.

7.3 Contingency Planning

Contingency planning was a significant component of the preparation leading up to the Games. There had to be a plan if an emergency resulted in the partial or full shutdown of the OSPC. If forecasts and/or alerts could not be produced by the OSPC, then contingencies would transfer operations to other Storm Prediction Centres in Canada. These contingencies are in addition to the normal operations of the MSC and its Storm Prediction Centres. As a contingency, the Storm Prediction Centre in Montréal, Quebec, was designated as the backup for the OSPC for marine forecasting and alerting programs, including production of the nearshore forecasts and alerts required for the Games.

In the event of a shut-down of the OSPC, the Prairie and Arctic Storm Prediction Centre (PASPC) in Winnipeg would operate the normal Ontario operations forecast and alerting program and assume the special requirements of the Games. The office was outfitted with a dedicated IPAW configured to view Ontario data and issue Ontario forecasts and alerts. If the outage in the OSPC operations was lengthy, forecasters would be transferred from the Toronto Office to Winnipeg and flown home once the situation in Ontario had returned to normal and was deemed to be stable.

The PASPC has two offices, one in Winnipeg and the other in Edmonton, Alberta. As part of the MSC’s normal contingency operations, the Winnipeg office backs up the OSPC while the Edmonton office backs up Winnipeg and is a second backup for the OSPC. In a sense, there was a double redundancy for the issuance of meteorological products and services for the 2015 Games.

Specific training was developed and delivered to train meteorologists in the contingency offices, in the event that the forecast and alerting program for the Games was transferred to their offices.

This contingency system is designed to be seamless, such that no one would notice that there was a disruption in service. This contingency operates 24/7/365, and fortunately, these detailed contingencies were never executed, though they were tested up to but not including flying forecasters out to Winnipeg. The testing included transferring systems across to the Winnipeg and Montréal offices, and the transition was seamless. It was equally smooth to have operations revert back to the OSPC once the testing was completed and deemed successful. The timing of the transition and the efficiency of it made for an effective test of the contingency plan for both the Games and for normal operations testing as well.

7.4 Emergency Preparedness

Over a year and a half prior to the Games, the Project Teams planned and prepared for the possibility of emergencies that could affect their operations during the Games. It was imperative that intricate testing be undertaken in a variety of formats to ensure that both staff and systems were operationally ready for the Games. These formats ranged from planning and running ECCC’s table top exercises to participating in table top exercises by other organizations. It included contingency testing, integration testing, systems tests and preparing staff. As our services were deemed to be mission-critical to the Games, there was a requirement for them to be available 24/7.

7.4.1 Table Top Exercise – Scenario Testing

In May 2014, under the leadership of the Implementation Team, ECCC invited representatives from Shared Services Canada, Public Safety Canada, Health Canada, the provincial Ministry of Health and Long Term Care, and the TO2015 organizing team to attend a table top exercise. The purpose of the exercise was to prepare ECCC and its partners for any eventualities that could, if they occurred, inhibit or impair the operation of the Project’s mission. This exercise was the first step of many to review and examine the risks and vulnerabilities associated with the operations and set-up of the Project in support of the Games. There was specific focus on the incident and command structure of the Project, and the effectiveness and efficiency of the communications in each scenario was examined. In particular, the notification and escalation procedures were key components.

There were three scenarios tested for this full-day exercise. The first was a scenario where a severe storm damaged some atmospheric monitoring equipment in the Mesonet, resulting in data outage and requiring repair. The objective of this exercise was to determine the time it would take for the weather watch and weather warning to impact decisions at the MOC and to determine the time to repair the damaged equipment. In the second scenario, a sudden IT outage affected the OSPC and required the forecast operations to be transferred to the Winnipeg Storm Prediction Centre. This scenario required Shared Services Canada to quickly diagnose and resolve the outage with minimal downtime. The objective was to maintain seamless service delivery to the TO2015 client while information would continue to flow to the briefers through the Winnipeg office, rather than the normal route from the OSPC. Once the IT service was restored, the transfer back to operations in Ontario was expected to be seamless as well. The third and final scenario tested the operations of service delivery to the client in the event of a communications outage between the briefers and the OSPC. In this case, the briefers would have to work with the latest information that was available to them and to find ways of continuing to receive that information in the event that they were cut off from communications due to external infrastructure failings (i.e., within the City of Toronto) and not from within our own network. There was a measure for the diagnosis time as well as for the development of contingency planning for this type of high-impact but low-probability situation. This exercise was developed to test multiple levels of failure and increased in complexity as the situation unfolded. Although the scenario was considered to be highly unlikely, it did allow all participants to consider future contingency plans in a different light.

The overall findings of the table top exercise clearly demonstrated ECCC’s level of preparedness and excellence with regard to dealing with emergencies and outages as a result of its normal course of business. ECCC has experience with alerting, notifying the public, escalating issues and resuming business quickly upon resolving failures, and the table top exercise showed these strengths.

The most obvious and anticipated risks relating specifically to the Games were found to be in the realm of communication linkages with our new partner organizations, learning each other’s lines of business and how we should and would work together for issue resolution and escalation as required. Recommendations were made for further meetings to decide on operating and escalating procedures and to have contact information with many levels of backup should outages become complex.

7.4.2 Table Top Exercise – Escalation and Notification

In June 2015, ECCC prepared another table top exercise solely based on Escalation and Notification procedures and the testing thereof. The purpose of the table top exercise was to prepare the ECCC for any eventualities that could, if they occurred, inhibit or impair the operation of the Project’s mission. This exercise focused on the incident and command structure of the Project and examined the effectiveness and efficiency of the communications in each scenario. In particular, the notification and escalation procedures were key components. The interplay between the Ontario Storm Prediction Centre forecasters and ECCC briefers was also tested. For this exercise, the participants were internal to the Department. One invitee from Public Safety Canada helped plan the day and attended as an observer who provided valuable feedback at the end of the day.

Figure 14.

Photo of Environment and Climate Change Canada’s Project Lead presenting to audience (twelve shown seated in the photo) at table top exercise for escalation and notification: June 12, 2015.
Photo: © Hong Lin

Similar scenarios were run in comparison to the table top exercise the year before. This time, the focus was much more on the communication between the OSPC and the briefers, and on how the briefers would present the information to their respective clients. It was imperative for the forecasters to know that they were no longer the final providers of products and services and that the two briefing teams would tailor the information to client-specific needs.

The findings of this exercise demonstrated that there was substantial improvement from the previous table top exercise the year before. The scenarios ran more smoothly, with individuals having a clearer picture of with whom they would communicate and what they would be required to say. Partners were more closely engaged by this point, and the risks that were identified the year before with regard to communication issues between ECCC and its partner and client organizations were mitigated by having closer contact with all of them.

7.4.3 Provincial Emergency Preparedness Exercise

As the provincial Government of Ontario was the official lead for safety and security for the Games, they were also responsible for preparatory exercises to test supporting federal, provincial and municipal governments along with the TO2015 organizing team. ECCC participated as the federal lead responsible for environmental emergencies. For three days in April 2015, the network of participants and players operated as though the scenarios in the script were real. The “injections,” as they are called, were arriving in what would be perceived as real time, and the PEOC played a central role in coordinating responses as quickly as organizations were able to determine their course of action.

The final report prepared by the province indicated that the complexities of these situations increased with the number of organizations required to resolve them. In every case, the issues and problems were resolved, with leadership coming from the appropriate organizations that engaged and mobilized appropriate partners.

7.4.4 Federal Table Top Exercise

In June 2015, the federal departments that had a role in the safety and security of the Games participated in a half-day table top exercise led by Public Safety Canada. A few high-impact scenarios had been selected for the exercise as situations that could potentially occur during the Games. These were discussed with each of the departmental representatives sharing their state of readiness in response to each scenario. This was an informal way of verifying and validating that each federal entity was ready for situations that would have impacts across departments. The organizations were being tested for their nimbleness, their ability to discover the complexities within a problem and their ability to engage with other organizations to resolve the situation. In addition, awareness was raised of the importance to communicate with the Government Operations Centre in Ottawa, Ontario, in order to co-facilitate the engagement of key departments and organizations in an attempt to mobilize the right resources to eliminate the problem and return operations to normal.

Date modified: