Evaluation of the Environmental Emergencies Program

March 2008

Report Clearance Steps

Planning phase completed June 2006
Report sent for management response October 2007
Management response received December 2007
Report completed January 2008
Report approved by Departmental Audit and
Evaluation Committee (DAEC)March 2008

Download in Adobe PDF format [679 KB]

Acronyms used in the report

CBRN

Chemical, Biological, Radiological and Nuclear

CCME

Canadian Council of Ministers of the Environment

CEPA

Canadian Environmental Protection Act, 1999

CESF

Competitiveness and Environmental Sustainability Framework

CNOPB

Canada-Newfoundland Offshore Petroleum Board

CNSOPB

Canada-Nova Scotia Offshore Petroleum Board

DAEC

Departmental Audit and Evaluation Committee

DMS

Departmental Management Services

E2

environmental emergencies

E2 Regulations

Environmental Emergency Regulations

EC

Environment Canada

EDF

Environmental Damages Funds

EEP

Environmental Emergencies Program

EP

Environmental Protection

ESTD

Emergencies Science and Technology Division

FNEP

Federal Nuclear Emergency Plan

FTE

Full-time equivalent (staff)

MOU

Memorandum of understanding

MSC

Meteorological Service of Canada

NCTP

National Counter-Terrorism Plan

NEEC

National Environmental Emergencies Centre

NEECP

National Environmental Emergencies Contingency Plan

NEES

National Environmental Emergency System

NEMISIS

National Enforcement Management Information System and Intelligence System

O&M

Operations and Maintenance

OPC

Outcome Project Components

OPG

Outcome Project Grouping

OPP

Outcome Project Plan

PCB

Polychlorinated biphenyls

PIA

Post-Incident Assessment

PSAT

Public Security and Anti-Terrorism

QMS

Quality Management System

R&D

Research and development

REET

Regional Environmental Emergencies Team

RIAS

Regulatory Impact Analysis Statement

RMAF

Results-based Management and Accountability Framework

S&T

Science and technology

SOP

Standard Operating Procedure

Y2K

Year 2000

Acknowledgments

The Evaluation Project Team, including Shelley Tice, Janet King, Lindsay Fitzpatrick, Karine Kisilenko, and Michelle Guertin and led by Robert Tkaczyk and V. Neimanis under the direction of the Director, Shelley Borys, would like to thank those individuals who contributed to this project and particularly all interviewees who provided insights and comments crucial to this evaluation.

Prepared by the Evaluation Division, Audit and Evaluation Branch

Table of Contents

• EXECUTIVE SUMMARY
• 1. INTRODUCTION
• 2. BACKGROUND and CONTEXT
  • 2.1 Background
  • 2.2 Mandate and Mission
  • 2.3 Program Delivery
    • 2.3.1 Prevention
    • 2.3.2 Preparedness
    • 2.3.3 Response
    • 2.3.4 Recovery
    • 2.3.5 Research and Development
  • 2.4 Program Logic Model
  • 2.5 Program Financial Resources
• 3. EVALUATION DESIGN
  • 3.1 Purpose and Scope
  • 3.2 Evaluation Approach and Methodology
• 4. FINDINGS
  • 4.1 Relevance
    • 4.1.1 Summary Points for Relevance
  • 4.2 Success
    • 4.2.1 Summary Points for Success
  • 4.3 Cost-Effectiveness
    • 4.3.1 Summary Points for Cost-effectiveness
  • 4.4 Design and Delivery
    • 4.4.1 Summary Points for Design and Delivery
  • 4.5 Stakeholder Opinions of Overall Strengths, Weaknesses, and Areas of Improvement
• 5. CONCLUSIONS
• 6. RECOMMENDATIONS
• 7. MANAGEMENT RESPONSE
  
  
• Annex 1 Evaluation Issues and Questions
• Annex 2 List of Background and Supporting Documentation
• Annex 3 List of Interviewees and Interview Guides
• Annex 4 Summary List of Ratings by Question

EXECUTIVE SUMMARY

Environment Canada's (EC) Audit and Evaluation Branch completed the evaluation of the Environmental Emergencies Program (EEP) in August 2007.

The evaluation of the Environmental Emergencies Program was included in Environment Canada's Audit and Evaluation Plan for 2005-2006¹ because:

• the Program has been in operation since 1973 and has never been evaluated;
• the Program was ranked as high risk in the course of the Environment Canada annual Audit and Evaluation risk assessment for the identification of audit and evaluation engagements because the Program is highly partnered and highly visible and it involves health and safety issues; and
• the Environmental Emergency Regulations, which are a component of the Program, are related to Public Security and Anti-Terrorism (PSAT) funds and are associated with Treasury Board evaluation requirements.

The purpose of the evaluation of the Environmental Emergencies Program was to examine whether the Program is:

• aligned to departmental strategic outcomes and addresses an actual need (relevance);
• achieving its intended outcomes (success), with a focus on immediate outcomes (in order to capture all the Program activity areas) while assessing the achievement of the intermediate and long-term outcomes of the Program;
• using the most appropriate and efficient means to achieve the outcomes (cost-effectiveness); and
• being delivered in the best possible way (design and delivery).

The evaluation of the Program was chosen as a pilot project to test a generic evaluation framework, approved by the Departmental Audit and Evaluation Committee (DAEC), composed of questions along with associated indicators and data sources for each of the evaluation issue areas of relevance, success, cost-effectiveness, and design and delivery. The evaluation addressed the 30 evaluation questions from this generic evaluation framework and involved the use of the following multiple lines of evidence:

• a review and analysis of documents related to the Program (where applicable, certain topics were researched in-depth, and these are identified as "research pieces");
• a review and analysis of a sample of Program activity files;
• telephone and in-person interviews using structured interview guides with internal Environment Canada staff and senior management as well as external stakeholders;
• a telephone survey of regulated facilities; and
• an independent review of the draft evaluation conclusions and recommendations by four experts.

The time frame under investigation was the period between July 2004 (development of the Results-based Management and Accountability Framework (RMAF)) and December 2006, and all five activity areas of the Environmental Emergencies Program (prevention, preparedness, response, recovery, and research and development) were included in the evaluation.

Below is an overview of the findings according to the four main issues -- relevance, success, cost-effectiveness, and design and delivery.

Relevance: Does the Program remain consistent with and contribute to the federal government's priorities and does it address actual needs?

1. There is evidence showing that the Program does serve a public interest.
2. The Program aligns well with Environment Canada's strategic outcomes.
3. There is evidence of a clear role for the government. What is less clear is the exact role that each level of government should undertake in environmental emergency management.

Success: Has the Program achieved its intended outcomes?

4. Overall, the Program is achieving the majority of the intended outcomes identified by the Environmental Emergencies Program and the Public Security and Anti-Terrorism (PSAT) initiative. Program success, however, is impeded by several design and delivery elements. Specifically, in those cases where outcomes are not being achieved, it is not clear, given the absence of an adequate performance measurement system, which specific activities are contributing to the achievement of the intended outcomes. This design and delivery flaw is also illustrated in the logic model. There is also the lack of integration of PSAT activities and outcomes into the prevention and preparedness elements of the Program and into the Program's logic model and performance measurement strategy. Further, there is the lack of a comprehensive impact on client awareness and the lack of a comprehensive assessment of the Program's capacity.

Cost-effectiveness: Are the most appropriate, cost-effective, and efficient means being used to achieve outcomes?

5. Key informant opinion emphasized the value of the Program. In the absence of comprehensive performance measurement reporting, it was not possible to determine cost-effectiveness/value for money objectively.
6. There is general agreement among those interviewed that the Program should include a cost-recovery element. What is less clear is which elements of the Program should be covered by public funds and to what extent.
7. Some alternative delivery methods have been examined but have not been extensively explored.

Design and Delivery: Is the Program designed and delivered in the best possible way?

8. In those cases where outcomes are not being achieved, it is not clear, given the absence of an adequate performance measurement system, which specific activities are contributing to the achievement of the intended outcomes. This design and delivery flaw is also illustrated in the logic model. There also is the lack of integration of Public Security and Anti-Terrorism activities and outcomes into the prevention and preparedness elements of the Program and into the Program's logic model and performance measurement strategy.
9. There is a lack of a comprehensive and robust outreach strategy to ensure that the intended impact on client awareness is achieved, specifically, in the activity areas of prevention and preparedness.
10. The Program's awareness of who constitutes the regulated community is incomplete (i.e., what industry sectors are included, along with their relevant information).
11. A comprehensive procedure to verify information supplied by clients concerning compliance with Environmental Emergency Regulations has not been implemented.
12. Although some elements of capacity analysis have been undertaken, there has been no complete and comprehensive assessment of the Program's capacity.
13. The performance measurement system within the Program is inadequate. Specifically, there are issues with data collection and data integrity; a lack of identified indicators for outcomes under the Public Security and Anti-Terrorism initiative; and limited baseline data for outcomes.
14. The division of roles and responsibilities within Environment Canada with respect to compliance promotion, as well as enforcement, are not clear, well understood, or properly documented.

This is the first evaluation examining the Environmental Emergencies Program, one of the more mature program areas within Environment Canada. Generally, the evaluation of the Program is positive: the Program is delivering a recognized public good, it is delivering on the Department's mandate, and no fundamental problems were found. Presented below are the main conclusions of the evaluation.

1. The Program continues to be relevant as it does serve the public interest and there is a clear role for government; nevertheless, there is an opportunity to further redefine the precise role for each level of government while taking into consideration the federal government's legislated responsibilities.
2. The Program is generally successful in meeting its intended outcomes. However, certain design and delivery elements limit quantitative measurement of the degree of its success.
3. Although qualitative evidence suggests that the Program is cost-effective, this could not be substantiated quantitatively.
4. There are a number of areas that need to be addressed in the design and delivery of the Program. These include the implementation of an effective performance measurement system; a comprehensive assessment of the Program's capacity; updating the Program's logic model and performance measurement strategy by integrating the Public Security and Anti-Terrorism activities and outcomes into the prevention and preparedness elements of the Program; and the implementation of a comprehensive and robust outreach strategy.
5. Roles and responsibilities in the areas of compliance promotion and enforcement in the context of the Program need to be clarified internally within Environment Canada.

Although areas of concern were raised in the evaluation with regard to the issues of relevance, success, and cost-effectiveness, these concerns are related to design and delivery aspects, and can therefore be addressed through adjustments in the design and delivery of the Program.

Fundamentally, the relevance of the Program and whether Environment Canada should continue to carry out this function are not in question, and this forms a firm basis upon which the Program operates.

Given current pressures on the Program, known capacity issues (including succession), and ongoing financial pressures at the government level, it is urgent that response to the recommendations be undertaken. The recommendations are presented in sequence, based on logical structure and time frame in which they should be addressed.

Recommendation #1:

Presently, within the department there are specific groups with functional responsibility for enforcement and for compliance promotion activities. Simultaneously, the EEP has functional responsibility for these two activities in the context of environmental emergencies. This has led to some confusion internally as to where the functional responsibilities begin and end for these two activities. This concern was raised by several internal and external key informant interview groups.

The evaluation also found that a number of areas need to be addressed with respect to the design and delivery of the Program:

• in the areas where Program outcomes are not being achieved, it is not clear, given the absence of an adequate performance measurement system, which specific activities are contributing to the achievement of the intended outcomes (this also applies to the logic model);
• a comprehensive assessment of the Program's capacity needs to be carried out;
• the Program's impact on client awareness needs to be comprehensive;
• a comprehensive procedure to verify information supplied by clients concerning compliance with the Environmental Emergency Regulations needs to be implemented; and
• the Program's logic model and performance measurement strategy need to be updated by integrating the Public Security and Anti-Terrorism activities and outcomes into the prevention and preparedness activity areas of the Program.

The EP Board should undertake to address the following recommendations in the order presented:

1. in collaboration with the Chief Enforcement Officer, clarify and adjust, where necessary, roles and responsibilities in the context of the Program, with a particular focus on compliance promotion and enforcement.
2. conduct further research regarding the activities related to raising awareness in the prevention and preparedness areas of the Program to determine which of these activities are not contributing to the achievement of these outcomes.
3. conduct a comprehensive program capacity assessment (i.e., needed vs. actual); where gaps emerge determine risks and implement mitigation strategies.
4. finalize and implement a performance measurement and reporting system.
5. address the Program's monitoring gaps (i.e., verification; client knowledge and outreach strategies).

Previously, environmental emergency planning and management, enforcement, and compliance promotion activities related to the Canadian Environmental Protection Act, 1999 and the Fisheries Act were located under the same Outcome Project Grouping (OPG) under the Environmental Protection (EP) Board. Thus, there was regular communication among the Environmental Emergencies Program, Enforcement, and Compliance Promotion and awareness of each other's activities. A decision was made in April 2006 to consolidate all enforcement activities within the department under the Departmental Management Services (DMS) Board. Presently, the fact that these activities are no longer consolidated under a single governance structure has exacerbated the confusion as to where functional responsibilities begin and end for these activities.

The activities and expected outcomes of the Environmental Emergencies Program need to be aligned with clearly defined roles and responsibilities. This may necessitate a revision of the Program's logic model. This would also provide an opportunity to update the Program's logic model and performance measurement strategy by integrating the Public Security and Anti-Terrorism activities and outcomes into the prevention and preparedness activity areas of the Program. The mandate of the evaluation was to examine the four evaluation issues (relevance, success, cost-effectiveness, and design and delivery) across the five activity areas of the Program. In those areas of the Program where outcomes were not fully met (e.g., awareness), the evaluation was not able to examine in greater depth which specific activities were or were not successful, because no adequate performance measurement system for the Program has been implemented. Building upon the findings of this evaluation, therefore, further research is necessary in these areas to identify which specific activities contributed to the achievement of these outcomes, with the objective of identifying areas to be improved, thereby enhancing the Program's efficiency and effectiveness. Part of this research may necessitate the collection of data concerning the specific activities in question to determine to what extent these activities were successful. Once the Program has identified which awareness activities have been successful, the Program should then develop and implement a comprehensive outreach strategy.

A comprehensive assessment of the Program's capacity needs to be conducted to identify gaps, if any, between available and needed capacity. If gaps are identified, related risks should be determined and mitigation strategies should be implemented to address these risks.

The Program should finalize and implement a performance measurement and reporting system, identifying key indicators and the data necessary to measure the Program's success in achieving its stated objectives.

The Program has specific monitoring gaps in the area of compliance that need to be examined and addressed to ensure proper linkages with the population on which it principally has an impact.

Management Response

Recommendation 1a:
EP Board agrees with this recommendation. Enforcement, regulatory compliance promotion and the provision of scientific and technical information and advice are three distinct departmental functions within Environment Canada. For legal as well as administrative reasons it is imperative that staff roles and responsibilities are clearly defined and understood both internally within the departments as well as by external stakeholders with whom such staff interact.

EP Board's delegated representative and the Chief Enforcement Officer have agreed to work towards development, by March 2008, of a governance document to clarify the relationship between emergency, compliance promotion and enforcement activities. This document will be developed according to Quality Management System (QMS) methodology.

Recommendation 1b:
EP Board agrees with this recommendation. With the exception of compliance promotion for the Environmental Emergency Regulation under Part 8 of CEPA 1999, the Department's historical approach to raising the awareness of our external stakeholders has, to a large extent, been to take advantage of regional and local initiatives led by other governments and OFGDs or private sector stakeholders. This approach has resulted in varying degrees of success.

EP Board commits to carry out a study that will identify key target audiences and assess the strengths, weaknesses and costs associated with a number of delivery mechanism options for raising emergency prevention and preparedness awareness. This study will be completed by June, 2008.

Recommendation 1c:
EP Board agrees with this recommendation. Significant work has been done in the past in determining resource needs required to effectively deliver the mandated responsibilities of the Environmental Emergencies Program. EP Board will draw upon this previous work to produce a comprehensive program wide assessment of capacity requirements, gaps, risks and mitigation strategies in relationship to Environment Canada's mandated roles and responsibilities. This assessment will also factor in the results of the awareness study which is to be undertaken in response to Recommendations #2 above and set forth options for management consideration. This work will be completed by January, 2009 so as to be ready in time for the FY 09/10 planning process.

Recommendation 1d:
EP Board agrees with this recommendation. Significant work has been carried out over the past several years in identifying a comprehensive set of performance indicators for the prevention, preparedness, response, recovery and R&D components of the Environmental Emergencies Program. Effort must now be directed towards analysing and selecting a smaller representative core set of indicators that will be effective in measuring program outcomes and effectiveness and which can be implemented in a nationally consistent and affordable manner. This work will be completed in parallel with the comprehensive program capacity assessment referenced above in response to Recommendation #3 and as such will be completed by January, 2009.

Recommendation 1e:
EP Board agrees with this recommendation and considers the development of an effective as well as affordable performance measurement strategy, as referenced above in response to Recommendation #4, as being critical to addressing the concerns identified in the evaluation of the Program.

1.0 INTRODUCTION

Environment Canada's (EC) Audit and Evaluation Branch completed the evaluation of the Environmental Emergencies Program (EEP) in August 2007.

The evaluation of the Environmental Emergencies Program was included in Environment Canada's Audit and Evaluation Plan for 2005-2006¹ because:

• the Program has been in operation since 1973 and has never been evaluated;
• the Program was ranked as high risk in the course of the Environment Canada annual Audit and Evaluation risk assessment for the identification of audit and evaluation engagements because the Program is highly partnered and highly visible and it involves health and safety issues; and
• the Environmental Emergency Regulations, which are a component of the Program, are related to Public Security and Anti-Terrorism (PSAT) funds and are associated with Treasury Board evaluation requirements.

The Environmental Emergencies Program is one of the more mature program areas within Environment Canada, having derived its original mandate from a 1973 Cabinet directive. The Program is structured into five activity areas: prevention, preparedness, response, recovery, and research and development. The mission of the Program is to reduce the frequency, severity, and consequences of environmental emergencies by promoting prevention of and preparedness for environmental emergencies, by providing response and recovery advice, and by advancing emergency science and technology.

The evaluation of the Program was also chosen as a pilot project to test an approved generic evaluation framework.

An evaluation committee was created with a mandate to facilitate and guide the evaluation process at the working level from start to finish. This committee was composed of representatives from the Audit and Evaluation Branch and the Environmental Emergencies Program.

This document presents the findings and recommendations of the evaluation of the Environmental Emergencies Program. It is organized in the following way:

• section 2.0 provides background information on the Program;
• section 3.1 outlines the purpose and scope of this evaluation;
• section 3.2 outlines the evaluation's approach and methodology;
• section 4.0 presents the evaluation's findings;
• sections 5.0 and 6.0 lay out, respectively, the conclusions and recommendations; and
• section 7.0 contains the management responses to the recommendations.

2.0 BACKGROUND and CONTEXT

2.1 Background

The Environmental Emergencies Program is one of the more mature program areas within Environment Canada, having derived its original mandate from a 1973 Cabinet directive. Over the years, the Program has tended to follow a course not unlike other event-driven programs, from being highly focussed and visible during major emergencies or disasters to being almost forgotten during periods between such events.

In the early 1970s, following a series of events, including the sinking of the Torrey Canyon (England, 1967) and the Arrow (Canada, 1970), the Environmental Emergencies Program was primarily oriented towards oil spills. Subsequent events, such as the wreck of the Amoco Cadiz (France, 1978), the Kurdistan (Canada, 1979), and the Exxon Valdez (United States, 1989), have reinforced a continuing need for prevention and preparedness to deal with oil spills.

In the 1980s, supported by a series of events, including the disasters in Bhopal, India, and Mexico City, Mexico (both in December 1984), where thousands of people were killed or injured, the need to focus on chemical and hazardous material releases was also identified. This was not so much a shift in emphasis in the Program as a realization that these types of events affect not only the environment but also the lives and health of people. The PCB spill along Highway 17 near Kenora, Ontario, in 1986, followed by the PCB storage fire in Saint-Basile-le-Grand, Quebec, in 1988 and two tire fires at Hagersville, Ontario, and St-Amable, Quebec, both in early 1990, demonstrated that there was a need to improve the Canadian emergency management system to deal with these types of events.

Since the early 1990s, several significant adjustments to the Program have had an impact on its direction. In late 1991, the Government of Canada announced Canada's Green Plan, which included $165 million in funding over five years. This funding included $40 million for Environmental Prediction and Warning, $100 million for the Marine Environmental Emergencies Response Strategy, and $25 million for the Hazardous Spills Prevention and Response Program. While only a portion of this federal funding was directed to Environment Canada, there was substantial growth in the Environmental Emergencies Program: the size of the Program and related expectations almost doubled over the next two to three years. In 1994, the Government of Canada announced Program Review and, in the decisions that followed, the budget of the Environmental Emergencies Program was reduced by 40%, from approximately $10 million to $6.25 million. A number of senior staff retired, and other personnel moved to various organizations. Beginning in 1997, the Environmental Emergencies Program began to re-examine itself in an attempt to define its responsibilities and capabilities better.

By 1999, the lead-up to the Y2K transition was beginning to have an impact on the course of the Program. Personnel were engaged in sectoral security reviews and in preparedness training to deal with any eventualities that might occur as a result of information technology malfunctions. The need for increased readiness was underlined by the attacks in the United States on the World Trade Center and the Pentagon on September 11, 2001. Environment Canada was actively engaged in the activities leading up to the federal budget of December 2001 (Securing Progress in an Uncertain World). New initiatives, such as the creation of a new Part 8 in the Canadian Environmental Protection Act, 1999 (CEPA) to deal with environmental matters related to emergencies, the development and implementation of the Environmental Emergency Regulations under section 200 of CEPA, and the Chemical, Biological, Radiological and Nuclear (CBRN) Research and Technology Initiative, have also shifted the focus of the Program into the area of public safety and security.

An important principle in understanding the roles and responsibilities encompassed within the Program is the concept of emergency management in Canada. Responsibility for managing environmental emergencies rests for the most part with the risk creator/potential polluter and subsequently with the level of government that has the specific legislated authority for environmental emergencies (municipal, provincial, and federal).

This escalation or hierarchical approach is inherent in the all-hazards approach to emergency management adopted in Canada. Several provincial and federal departments or agencies have legislated mandates to carry out and vital functions to perform in dealing with emergencies, and these respective roles must be recognized and respected. Industry (as a potential polluter) also has a critical role to play in all phases of emergency management. A major challenge for all parties is to ensure that arrangements and agreements are in place to ensure cooperation and coordination when the need arises.

2.2 Mandate and Mission

Part 8 of the Canadian Environmental Protection Act, 1999 defines an environmental emergency as "an uncontrolled, unplanned or accidental release, or release in contravention of regulations or interim orders made under this Part, of a substance² into the environment; or the reasonable likelihood of such a release into the environment" that has or may have "an immediate or long-term harmful effect on the environment, constitute[s] or may constitute a danger to the environment on which human life depends, or constitute[s] or may constitute a danger in Canada to human life or health." Environment Canada, being the lead environmental agency for the federal government, is responsible for the provision of essential services to prevent, prepare for, respond to, and recover from releases in order to protect the environment and human health from the threats of pollution resulting from environmental emergencies.

These essential services provided by Environment Canada are delivered through the Environmental Emergencies Program. The Program's mandate is derived from an assortment of federal legislation and policies, including:

• Constitution Act (1867);
• Cabinet directive (1973)³;
• Emergency Preparedness Act (1985);
• Fisheries Act (1985);
• Migratory Birds Convention Act, 1994;
• Federal Policy on Departmental Planning Responsibilities for Emergency Preparedness (1995); and
• Canadian Environmental Protection Act, 1999.

The mission of the Program is to reduce the frequency, severity, and consequences of environmental emergencies by promoting prevention of and preparedness for environmental emergencies, by providing response and recovery advice, and by advancing emergency science and technology.

In 2001, the Program's responsibilities increased due to the development of the Government of Canada Public Security and Anti-Terrorism strategy, which provided $7.8 billion over six years to various federal departments to enhance and implement security measures to protect the health and safety of Canadians. The Environmental Emergencies Program received funds to implement measures (the Environmental Emergency Regulations) that would improve the prevention of environmental emergencies, preparedness for them, response to them, and recovery from them at facilities that manage substances that, if released accidentally or by deliberate action, would endanger human health or environmental quality.

2.3 Program Delivery

The Environmental Emergencies Program is structured into five activity areas: prevention, preparedness, response, recovery, and research and development.

2.3.1 Prevention

The activities under the area of prevention include the administration, review, and management of the preparation and maintenance of environmental emergency plans, the identification of hazards, and the assessment of risks in order to develop and implement strategies to minimize the likelihood of an emergency. Activities also include the facilitation and delivery of training to stakeholders on the prevention of spills. The Environmental Emergency Regulations (2003) under the Canadian Environmental Protection Act, 1999 require anyone storing or using specified toxic or hazardous substances at or above specified thresholds to prepare and implement environmental emergency plans. These plans must provide information on the types of emergencies caused by toxic or other hazardous substances that might occur and the corresponding prevention, preparedness, response, and recovery information.

The primary responsibility for the prevention of environmental emergencies associated with the transportation of toxic or other hazardous materials rests with Transport Canada as it relates to road, rail, or marine modes of transport and with the National Energy Board as it relates to the interprovincial or international transportation of such materials by pipeline. In these cases, Environment Canada's role is that of an advocate for increased environmental protection.

2.3.2 Preparedness

Under the preparedness activity area, Environment Canada is responsible for coordinating both national and regional environmental emergency preparedness capabilities. This involves the development of contingency plans, such as the National Environmental Emergencies Contingency Plan, which outlines the responsibilities of federal departments in the event of an environmental emergency, procedures on how to respond to environmental emergencies, and training standards for departmental environmental emergencies officers.

At the regional level, Environment Canada coordinates the Regional Environmental Emergencies Teams (REET), composed of representatives from federal, provincial, and municipal government agencies responsible for environmental protection; Aboriginal representatives; and experts from industry and non-governmental organizations. The Regional Environmental Emergencies Teams (REETs) provide consolidated expert advice during environmental emergencies.

Furthermore, Environment Canada provides leadership and guidance to departments, provinces, agencies, and industry in the development of contingency plans, spill response preparedness plans, and reporting and response systems; participates in environmental emergencies exercises; and maintains networks with industry, other government departments, and response organizations.

2.3.3 Response

During environmental emergencies, Environment Canada is responsible for federal government oversight of response actions taken by the responsible party or parties; the provision of scientific and technical advice on weather, sea state, and the fate, behaviour, and effects of chemicals; sampling and analysis; countermeasures; and sensitivity mapping and trajectory modelling for some on-scene operations in order to minimize environmental damage. Furthermore, Environment Canada operates the 24/7 National Environmental Emergencies Centre (NEEC). Depending on the nature and the severity of the emergency, the National Environmental Emergencies Centre helps coordinate information and resource needs, tracks response progress, and provides updates for senior management at Environment Canada.

2.3.4 Recovery

After an environmental emergency occurs, Environment Canada is responsible for ensuring that those responsible for the environmental emergency have properly assessed and, to the maximum extent feasible, restored any environmental damage caused by the incident, by undertaking or coordinating restoration where required and recovering costs from polluters. This involves the development and application of damage assessment and restoration tools and techniques and assistance in the management of the Environmental Damages Fund, which serves as a federal government trust account for managing and disbursing environmental compensation payments.

2.3.5 Research and Development

The research and development (R&D) activity area, mainly operated from the Environmental Science and Technology Centre (ESTC), involves the development of spill models, analysis methods, fate and behaviour algorithms, measurement and remote sensing capabilities, decontamination protocols, and countermeasures used during incidents. The Meteorological Service of Canada (MSC) also develops and delivers weather predictions and modelling results.

Another component in this area involves ongoing development of prevention technologies, tools, and approaches.

2.4 Program Logic Model

The Results-based Management and Accountability Framework (RMAF), which was developed in July 2004, includes a summary logic model of the Environmental Emergencies Program (Figure 1) that identifies the linkages between the Program's activities and the achievement of its outcomes.

The logic model, however, does not encompass the functions of the Program associated with the Public Security and Anti-Terrorism strategy. The Program's responsibilities increased due to the development in 2001 of the strategy, which provided funds to enhance and implement security measures to protect the health and safety of Canadians. The Program did identify specific outcomes⁴ related to the Environmental Emergency Regulations, and these are documented in the annual Public Security and Anti-Terrorism reports to Treasury Board.

The Public Security and Anti-Terrorism Intermediate Outcome was identified as:

• improved environmental emergency management at facilities and communities, including product substitution to less hazardous substances, prevention, preparedness, response, and recovery.

The Public Security and Anti-Terrorism Immediate Outcomes were identified as:

• increased awareness of the Environmental Emergency Regulations;
• increased EC's awareness of the regulated community;
• increased community awareness of the Environmental Emergency Regulations and of surrounding risks;
• increased community participation in environmental emergency plan preparation;
• increased partnerships/relationships between government and industry; and
• increased compliance with the Environmental Emergency Regulations.

Figure 1: Summary Logic Model for the Environmental Emergencies Program

Click to enlarge

2.5 Program Financial Resources

Table 1 provides a summary of the Environmental Emergencies Program's budget allocation for salary and for operations and maintenance (O&M) by region (including a separate column for the Emergencies Science and Technology Division (ESTD)) and by year, for fiscal years 2002-2003 to 2005-2006. These figures include the funding allocated to the Program under the Public Security and Anti-Terrorism strategy.

Table 1: Budget Allocation for Salary and Operations and Maintenance (O&M) for the Environmental Emergencies Program

3.0 EVALUATION DESIGN

3.1 Purpose and Scope

The purpose of the evaluation of the Environmental Emergencies Program was to examine whether the Program is:

• aligned to departmental strategic outcomes and addresses an actual need (relevance);
• achieving its intended outcomes (success), with a focus on immediate outcomes (in order to capture all the Program activity areas) while assessing the achievement of the intermediate and long-term outcomes of the Program;
• using the most appropriate and efficient means to achieve the outcomes (cost-effectiveness); and
• being delivered in the best possible way (design and delivery).

Related to the evaluation issue of success, Table 2 presents the specific intended Program outcomes. Immediate outcomes relate to each of the Program's five activity areas and to activities related to the Public Security and Anti-Terrorism strategy. Intermediate outcomes relate to the Program overall and to activities related to the Public Security and Anti-Terrorism strategy. Finally, the long-term outcome relates to the Program overall.

Table 2: Environmental Emergencies Program Intended Outcomes⁷

All five activity areas of the Environmental Emergencies Program (prevention, preparedness, response, recovery, and research and development) were included in the evaluation.

The time frame under investigation was the period between July 2004 (development of the Results-based Management and Accountability Framework) and December 2006.

3.2 Evaluation Approach and Methodology

The Departmental Audit and Evaluation Committee of Environment Canada approved an evaluation framework composed of questions along with associated indicators and data sources. The evaluation rigorously applied the 30 evaluation questions from the generic evaluation framework and examined all four evaluation issues (noted above).

The framework was built on a traditional logic model, built on the premise that one uses resources (Inputs) and does things (Activities) which result in products (Outputs) targeted at certain stakeholders (Reach) in order to achieve certain results, desired end-points, or altered conditions (Outcomes). Subsequently, a suite of generic evaluation questions (along with associated indicators and data sources) was developed that could be applied to any program and examined in tandem with the completed logic model. The questions focus on the four main evaluation issues -- relevance, success, cost-effectiveness, and design and delivery. These generic questions (which include the seven expenditure review questions) were coupled with a handful of supplementary evaluation questions pertaining to the specific program being evaluated. These supplementary questions were determined in the context of carrying out this evaluation. The evaluation framework is shown in Annex 1.

The evaluation involved the use of multiple lines of evidence. These methodological approaches are described in Table 3.

Table 3: Methodological Approaches

A number of limitations presented themselves during the course of this evaluation. The following represent the more noteworthy ones:

1. Certain planned methodological approaches could not be undertaken. Specifically, in considering the use of case studies (given the difficulties of integrating non-generalizable case study findings into an evaluation and the limited resources available), the evaluation committee concluded that greater value could be achieved by focussing the evaluation resources on the other methodologies.
2. The volume of data collected during the conducting phase of the evaluation (stemming from 99 interviews and the review of over 200 documents) slowed the report delivery.

4.0 FINDINGS

Below are the findings of this evaluation presented by the four issue categories-relevance, success, cost-effectiveness, and design and delivery -- and using the set of questions developed for the generic evaluation framework, as presented in Annex 1. For each finding, the number of interview and survey respondents, as well as the number of research pieces, is also provided.

4.1 Relevance

This section will examine the Program's relevance -- the degree of alignment with the directions and priorities of Environment Canada and clarity of the intent of the Program, as well as the continuing rationale for the Program, given prevailing areas of need and the potential for overlap with other initiatives.

Findings:

• Almost all stakeholders felt the Environmental Emergencies Program serves the public interest.
• Survey respondents from the regulated community felt that the Program serves the public interest to a moderate extent.

Findings:

• Environment Canada stakeholders identified a clear connection between the Environmental Emergencies Program and departmental strategic outcomes and departmental outcomes.

Findings:

• There is a clear role for government in dealing with environmental emergencies, although different levels of government play distinct and differing roles, depending on their legislated authority.
• The variety of roles includes providing technical expertise, providing coordination and facilitation of response to emergencies, and bringing together stakeholders and their individual interests.

Findings:

• Most external non-federal partners and senior management at Environment Canada feel that there is a role for the federal government in dealing with environmental emergencies; however, it is not completely clear what that role should be (the role of Environment Canada currently varies by region).
• Different levels of government play different roles in environmental emergencies; if the federal government were to realign its role with respect to environmental emergencies, the provinces and territories would be the most logical candidate to take on further responsibilities.

4.1.1 Summary Points for Relevance

1. There is evidence showing that the Environmental Emergencies Program does serve a public interest.
2. The Environmental Emergencies Program aligns well with Environment Canada's strategic outcomes.
3. There is evidence of a clear role for the government. What is less clear is the exact role that each level of government should undertake in environmental emergency management.

4.2 Success

This section will examine the results of the Program, both intended and unintended, in areas supporting the mandate of the Program.

Findings:

• Both interview opinions and the survey of regulated facilities indicated that there has been an increase in awareness of both the risks of environmental emergencies and associated mitigation actions.
• The interview results for both the external non-federal partners and the regulated community were divided. Some believed there has been a significant increase in awareness, while others indicated that the increase in awareness among stakeholders was only partial to marginal.
• The survey of regulated facilities revealed that respondents from over half of the facilities felt that there has been a moderate impact on targeted stakeholders' awareness of the nature and scope of risks of environmental emergencies as well as of appropriate mitigation measures. Yet close to 30% felt that there was little to no impact for either element.
• There is insufficient evidence to determine quantitatively whether there has been increased awareness, as there are no information or statistics available that indicate formal measurement of such elements over time.

Findings:

• Many of the stakeholders feel that clear and concise compliance and enforcement strategies do exist, although the regulated community, in general, felt it was difficult to comment on the achievement of the outcome. Some internal partner respondents noted a lack of evaluation of the implementation of environmental emergency plans and the need for clarification regarding roles and responsibilities surrounding compliance and enforcement within the department. A need for clarification regarding compliance with the Environmental Emergency Regulations (timing, documentation needed, application of compliance and enforcement) was mentioned by respondents from the regulated community.
• Of those surveyed, 25% could not comment on the Program's impact on the implementation of clear compliance and enforcement strategies with respect to the Fisheries Act. Among those with an opinion, most said the impact of the Program on implementation has been moderate, while fairly equal percentages indicated that the impact has been large, on the one hand, or small, on the other hand. With respect to the Environmental Emergency Regulations, 85% felt that the Program had at least a moderate impact.
• A document review revealed that clear and concise compliance and enforcement strategies exist for section 200 of the Canadian Environmental Protection Act, 1999 (including guidelines and tools), as well as for the habitat protection and pollution prevention provisions of the Fisheries Act.

Findings:

• Most stakeholders feel that the Program is ready to provide timely, relevant scientific and technical advice and support consistent with mandated responsibilities. Positive examples included Regional Environmental Emergencies Teams (REETs) and overall knowledgeable staff. On the negative side, the issue of limited resources and their effect on the Program's ability to continue to maintain its present level of readiness, the need to update Program documentation, and regional variability regarding level of preparedness were cited.
• Overall, evidence suggests that Environment Canada has appropriate strategies, plans (e.g., contingency) and activities (e.g., training; exercises) in place to provide timely, relevant scientific advice and support consistent with its mandated responsibilities. However, some of its documentation needs to be updated to reflect recent changes in the environmental emergency response infrastructure. In addition, a complete and comprehensive assessment of the Program's capacity needs to be undertaken to ensure that the strategies and activities can be implemented as planned.

Findings:

• Although interviews do point to some improvements in client awareness and readiness to manage environmental emergencies, respondents questioned the relationship between the activities of the Program and this outcome.
• Several respondents from the regulated community questioned whether the activities of the Program improved their level of preparedness.
• However, the survey results from the regulated facilities showed that respondents from over half of the facilities felt that there has been a moderate impact on both client awareness and readiness to manage environmental emergencies. Just over 30% felt that the impact of the Program on client awareness and readiness was to a great extent.
• There is insufficient evidence to determine quantitatively whether there has been increased awareness and readiness, as there are no information or statistics available that indicate formal measurement of such elements over time.

Findings:

• Stakeholders feel that this outcome has been successfully achieved; in fact, they see this area as a strength of the Program. The only concern raised was possible overlap in coordination with provincial response units.
• Positive examples of science and technology advice and support included the operation of Regional Environmental Emergencies Teams (REETs) and knowledgeable Program staff.
• Since 1992, the Emergencies Science and Technology Division has provided major response assistance, including data, modelling, analysis, and countermeasures, 300 times (46 incidents since 2004).
• This area still requires enhanced performance measures and reporting in order to allow this outcome to be monitored.

Findings:

• During the time period of July 2004 to December 2006, there were no instances where Environment Canada had the lead.
• Stakeholders had a difficult time responding to this question, as it was not clear to them what the lead role referred to and when Environment Canada would assume this role.
• All stakeholders commented that Environment Canada rarely has the lead in emergency situations; Environment Canada would take the lead only if an emergency occurred on federal land or in the event of a transboundary event or of a spill whose source could not be determined.
• In fact, it is the creator/polluter that takes the lead in an emergency and subsequently the level of government that has the specific legislated authority for that particular environmental emergency (municipal, provincial, federal). Where the federal government is required to intervene, a Minister may be named to assume the lead role and coordinate the collective effort of the federal government (e.g., Canadian Coast Guard, Transport Canada).
• For the majority of incidents, Environment Canada will provide support, including technical and scientific advice and monitoring and weather hazard forecasts, advisories, alerts, and warnings.

Findings:

• There is agreement in the survey and in interviews that there is increased awareness of the need for assessment and restoration of damages caused by environmental emergencies.
• The Program's external partners, as a consensus opinion, agreed that there has been an increase of awareness of the assessment and restoration of damages caused by environmental emergencies. Training, workshops, and the Environmental Damages Fund were all cited as helpful in achieving such increases.
• The survey of regulated facilities revealed that respondents from over half of the facilities felt that there has been a moderate impact on targeted stakeholders' awareness of the importance of assessing and restoring damages caused by environmental emergencies, while 30% felt that the impact was to a greater extent.
• There is, however, insufficient evidence to determine whether there has been increased awareness, as there are no information or statistics available that indicate formal measurement of such elements over time.

Findings:

• Key informant opinions as well as research evidence suggest that there has been advancement and development of scientific knowledge, technologies, tools, and approaches associated with environmental emergency prevention, preparedness, response, and recovery.
• Interviews with Program deliverers suggest advancement; internal partners cite positive cases yet note room for improvement, particularly in the broadening of the scope of research and development (e.g., decontamination of facilities, research alternatives to oil spill cleanup, policy implications of research and development findings).
• The majority of the two groups of external stakeholders, as well as the regulated community, also recognize the advancement in this area by Environment Canada, even citing the department as a world leader in spill research.
• Research and development organizations are overwhelmingly favourable in their opinion of the Program's success in developing environmental emergency research and development.
• In addition, documentation regarding the research and development process as well as research and development output products is available.

Findings:

• Many stakeholders feel that client implementation of emergency prevention, response, and recovery plans has been achieved to some extent, although internal and external (non-federal) partners found it difficult to comment on the achievement of the outcome as a whole. Federal partner respondents commented positively, but mentioned that this was an ongoing process. Program staff added that notices have been submitted regarding environmental emergency plans in order to ensure clients comply with regulations. However, difficulties were cited in attributing achievements to the Program and in measuring the degree and extent of achievement, especially given the lack of data/data collection (stakeholders mentioned other factors that may contribute to the achievement of this outcome, such as international conventions and agreements, the Canada Shipping Act, 2001, and public pressure).
• In total, 95% of those surveyed reported that their facilities have implemented environmental emergency plans to at least a moderate extent (69% felt it was to a tremendous extent). Almost 90% of those surveyed felt that the Program has had at least a moderate impact on client implementation of emergency prevention, response, and recovery plans and practices.
• A document review revealed that as of 2004-2005, 2100 facilities had been identified that needed to prepare and implement environmental emergency plans.
• There is a lack of information or statistics available that indicate formal measurement of such elements over time. Exercises have been conducted and do provide some indication of support of progress towards the achievement of the outcome. The Program has also begun to carry out reviews of environmental emergency plans and on-site visits (examples were provided from the Atlantic and Ontario regions).

Findings:

• For the purpose of this question, "investment" is interpreted to include activities and resources.
• The EC Program deliverers interviewed felt that investment in research and development by both government and industry in environmental emergency prevention and preparedness had decreased.
• Most of the other stakeholders that were interviewed could not comment on the level of investment and whether it had increased or decreased, although there was a general sense of an overall increase.
• Funding received from joint ventures/partnerships amounted to $2,681,552 in FY 2002-2003; $2,112,609 in FY 2003-2004; $2,768,776 in FY 2004-2005; and $3,569,442 in FY 2005-2006.
• A review of documentation revealed insufficient evidence to determine whether there has been increased investment in environmental emergency prevention and preparedness by industry and governments, as there are no information or statistics available that indicate formal measurement of such changes.

Findings:

• Just over half of interview respondents felt that there has been a reduction in the frequency, severity and consequences of environmental emergencies that affect Canada. However, stakeholders stated that it is difficult to quantify and measure the actual frequency, severity, and consequences of environmental emergencies, in part because of a lack of data/data integrity.
• Another issue that was identified was uncertainties of attribution; the relationship between the activities of the Program and this outcome are unclear.
• The following are examples of reasons why it is difficult to measure this outcome: changes in reporting requirements and laws, increased enforcement, application of preventive measures, and increased awareness of the consequences of environmental emergencies.
• The survey of regulated facilities found that just over half of respondents gave a moderate rating of the impact of the Program on the reduction in the frequency, severity, and consequences of environmental emergencies.
• Statistical data available from trend documents do not use the same parameters and therefore cannot be compared in any time series manner: spills numbers from 1974 to 1995 are available, and moderate and major spills are recorded from 2000 to the present.
• There is insufficient evidence to determine whether there has been a reduction in the frequency, severity, and consequences of environmental emergencies.

Findings:

• Interviewed stakeholders feel that the Program has led to a high degree of awareness of the Environmental Emergency Regulations; this has been achieved through consultations, work with associations, participation at conferences and workshops, and articles in magazines/journals.
• A survey of the regulated community found that 90% of respondents felt that awareness of the Environmental Emergency Regulations has been at least moderately affected by the Program (53% to a great/tremendous extent and 37% to a moderate extent).
• There is, however, insufficient evidence to determine whether there has been an increase in awareness generated by the Program, as there are no information or statistics available that indicate a level of awareness of the Environmental Emergency Regulations.

Findings:

• Interviewed stakeholders feel that the Program has led to an awareness of the regulated community; this has been achieved through numerous workshops throughout the country. It was noted, however, that more work needs to be done in this area.
• Internal partners had difficulty responding to this question.
• A document review revealed that although the number of facilities subject to the Environmental Emergency Regulations was initially estimated at 1500 facilities in 2003-2004, almost 3000 facilities were actually identified; of these, 1900 were determined to need environmental emergency plans. In 2004-2005, the number of identified facilities increased to more than 3000, with 2100 of these identified as needing environmental emergency plans. The same numbers were reported in 2005-2006.
• Although the level of Environment Canada's awareness of the regulated community has increased, there is insufficient evidence to determine whether the increase has been adequate, as the information available is incomplete.

Findings:

• In the few interviews conducted, most of the respondents from the regulated community felt that the community was not aware of the Environmental Emergency Regulations and surrounding risks. However, there was some confusion as to what is meant by "community": whether this includes individuals, associations, the general public, and/or those living in close proximity to a regulated plant.
• A survey of the regulated community revealed that approximately half of respondents felt the Program had a moderate impact on awareness in surrounding communities of the Environmental Emergency Regulations and surrounding risks, 25% of respondents felt there was a great impact, and the final quarter felt there was little to no impact.
• There is insufficient evidence to determine whether the Program has been responsible for an increase in community awareness, as there are no information or statistics available that indicate a level of awareness of the Environmental Emergency Regulations and surrounding risks.

Findings:

• There is some confusion as to what is meant by "community" (does this include individuals, associations, the general public, and/or those living in close proximity to a regulated plant?), thus making it difficult to assess the achievement of this outcome.
• In the few interviews conducted, the vast majority of the regulated community said that community participation in the preparation of environmental emergency plans has increased among stakeholder groups.
• The survey showed similar findings: approximately half of the respondents felt that the Program had a moderate effect on community participation in the preparation of environmental emergency plans, while the others were generally split between more positive and negative opinions.
• There is, however, insufficient evidence to determine whether there has been increased community partnerships in emergency plan preparation, as there are no information or statistics available that indicate formal measurement of such relationships.

Findings:

• Interviews with Environment Canada staff and management as well as the regulated community reveal that there is an increased awareness and an increase in partnerships/relationships between government and industry.
• The survey showed similar findings but was more tempered: over half of the respondents felt that the Program had a moderate effect on partnerships/relationships between government and industry, while the others were generally split between more positive and negative opinions.
• There is insufficient evidence to determine whether there has been increased partnership and relationship between government and industry, as there are no information or statistics available that indicate formal measurement of such relationships.

Findings:

• Compliance with a regulation is mandatory, and, in fact, the survey (self-reporting) reveals a high degree of national compliance (69% of facilities have implemented environmental emergency plans to a great extent, while 26% of facilities have implemented plans to a moderate extent). Only 3% or 11 out of 350 facilities reported little or no implementation of environmental emergency plans.
• Some variation in the extent of implementation is reported, depending on the region, the size of the facility, and the sector. While 69% of facilities have implemented their environmental emergency plans to a great extent, geographically, Quebec lags behind at 48%, as do those in smaller facilities (59%) and those in the agriculture sector (60%).
• There is also commentary on the level of effort required to implement the Environmental Emergency Regulations; nearly two thirds reported that as being moderately difficult.
• There is insufficient evidence to determine whether there has been an increase in compliance with the Environmental Emergency Regulations, as there are no information or statistics available.

Findings:

• The evidence suggests that there is both improved environmental emergency management and the substitution of less hazardous substances. However, there are differing views as to the extent to which these have been achieved.
• The majority of Program deliverers interviewed believe that this outcome has been achieved, but the evidence is anecdotal, or not formally reported. There is also recognition of the difficulty of measuring the degree of change.
• The majority of representatives from the regulated community interviewed indicated that the Program has contributed to improved environmental emergency management at facilities and in communities. Examples included product substitution and reduction in inventories.
• The vast majority of the regulated community interviewed also felt that the Program has contributed to management improvements.
• The survey of regulated facilities revealed an even greater degree of impact. Over 85% of respondents felt that improved environmental emergency management had occurred; only 15% said such impact was little or none. In terms of product substitutions, half of the representatives reported that the Environmental Emergency Regulations have not led to product substitutions at their facilities. In terms of product substitutions, oil and gas, chemical, and petroleum facilities were significantly more inclined than agriculture facilities to say that the Environmental Emergency Regulations have not at all led to product substitution.
• There is insufficient evidence to determine whether there has been improved management or product substitution, as there are no information or statistics available that indicate formal measurement of such changes.

Findings:

• Key informant interviews identify three negative external influences on the Program: the lack of expertise, resources, and capacity at the provincial level; the dependence of the Program on information and cooperation with the provinces; and the level of political attention given to environmental emergencies.
  • Capacity of provinces: Even though provinces are officially the lead agent for specific environmental emergencies, some provinces have fewer resources, which translate into an inability to prepare for and respond to emergencies when they occur. This lack of capacity requires that the Environmental Emergencies Program step in and provide those services.
  • Cooperation with provinces: Lack of engagement on the part of provinces can affect the ability of the Environmental Emergencies Program to respond to environmental emergencies.
  • Political attention: International incidents that receive a large amount of media coverage such as the Bhopal incident and 9/11 focus public attention on environmental emergencies and, as a result, increase the profile of the Program, lead to more rapid development of regulations, and stimulate institutional acceptance and preparedness on the part of government and industry stakeholders. However, when public attention is not present, political attention wanes and the Program loses visibility.
• The interviews also identify several negative internal influences on the Program, namely the fluctuation of the level of financial support for the Program and the subsequent impact on its ability to staff, as well as the transformation of Environment Canada in terms of accountability and decision making.

Findings:

• A majority of key informants across stakeholder groups were unable to identify unanticipated results of the Program. One respondent provided an example of a negative result. The Environmental Emergency Regulations have prompted industries to reduce the amounts of hazardous materials they keep on site, with the result that hazardous materials are being transported more frequently.
• Several positive unanticipated results were mentioned. In particular, information and tools generated by the Program are being used by other organizations for purposes other than those originally intended (e.g., use of geophysical technology (coastal mapping) to find debris from plane crashes). Also, key informants attributed to the Program increased partnership development and information sharing amongst stakeholders, as well as funding of the Environmental Damages Fund through the successful prosecution of environmental crimes. Outsourcing of some Program activities has also helped build private sector capacity in the environmental emergency field.
• Key informants did not provide answers to the question on how unanticipated results have been addressed.

4.2.1 Summary Points for Success

1. Overall, the Environmental Emergencies Program is achieving the majority of intended outcomes. Program success, however, is impeded by several design and delivery elements. Specifically, given the absence of an adequate performance measurement system, it is not clear which specific activities are contributing to or impeding the achievement of the intended outcomes. This design and delivery flaw is also illustrated in the logic model. There is also the lack of integration into the Program's logic model and performance measurement strategy (specifically, into the prevention and preparedness elements of the Program) of activities and outcomes under the Public Security and Anti-Terrorism strategy. Further, there is the lack of a comprehensive impact on client awareness, as well as the lack of a comprehensive assessment of the Program's capacity.

4.3 Cost-Effectiveness

This section will examine cost-effectiveness, including value for money, cost recovery, and alternative delivery methods, as a guide to determine the potential for the Program to be made more effective in the use of its resources and the pursuit of its mandate.

Findings:

• There is universal opinion across all interviews and the survey that there is value for money in the Program. What varies is the degree to which each key informant category feels the Program has achieved value for money. Interview respondents could not offer any comment regarding the relationship between money spent and results achieved.
• Environment Canada staff as well as management believed that the Program was cost-effective (due to extensive partnerships and leveraging of funding) but that there was room for improvement. For example, changes could be made in the variation of value for money between regions and the need for more field response, in terms of providing environmental advice and assistance. Respondents also commented on the public value of the Program, although there was uncertainty as to how one might quantify this value in financial terms.
• The majority of external partners felt the Program provided good value for money and protected an essential public good. The non-federal partners believed that great value for money was being achieved by a very necessary program for society. Questions of sufficient funding were raised.
• In the few interviews conducted, the regulated community felt that good value for money was offered but noted difficulty in determining how to value the avoidance of incidents. The survey of the regulated community revealed a moderate response regarding value for money, with the greatest value conferred by smaller companies.
• Financial information indicates a close correlation between planned and actual spending, which has been within 2% since 2003-2004.
• However, there are insufficient data to assess achievement of outcomes. This limitation prevents a complete analysis of cost-effectiveness, which is needed to determine value for money.

Findings:

• There is general agreement among those interviewed that the Program should include a cost-recovery element.
• What is less clear is the nature of the cost-recovery element.
  • Internal partners felt that the "polluter pays" model was a possibility but cautioned that accidental spills should not incur a cost to the polluter⁹; they also felt that there were opportunities to use the Environmental Damages Fund to recoup the costs associated with a spill.¹⁰
  • External partners are evenly divided on whether costs should be recovered; they feel the polluter should pay but don't want the level of service to be reduced (depends on service and circumstances, e.g., wouldn't want there to be a charge for prevention services).
• Opinion is also divided on whether cost recovery is currently in place and whether a formal cost-recovery policy/fee structure exists.
  • Program deliverers indicated that there was use of cost recovery, but that it depended on the activity area (e.g., research and development and science and technology advice and publications).
  • Program management indicated that costs are currently recovered through the Marine Spill Fund (although this is not consistently applied).
  • The regulated community reported that for the most part they do not pay for services provided by the Program.
  • A review of documents identified a cost-recovery element for:
    • Work provided to other government departments regarding science and technology research and development by the Emergencies Science and Technology Division (ESTD). The costs recovered in the last four years range from a low of $440,000 to a high of $940,000 annually in science and technology research and development.
    • the costs of providing response based on legislation, e.g., Canada Shipping Act, 2001, Canadian Environmental Protection Act, 1999 (as stated in a 1997 document).

Findings:

• There is a general feeling among the range of interviewees that the Program is employing effective delivery methods.
• Some alternatives have been or are being explored to increase standardization and to address capacity issues across regions (e.g., one-window notification). Other suggestions provided (although not alternative delivery mechanisms) include more formalized agreements that incorporate performance measures, improved Regional Environmental Emergencies Team (REET) plans, improved data collection and storage, improved communications, and a more coordinated approach.
• However, based on other findings, such as the lack of clarity of roles and responsibilities of the federal government vis-à-vis other levels of government/private sector, there is limited evidence of consideration of alternative delivery methods.

4.3.1 Summary Points for Cost-effectiveness

1. Key informant opinion emphasized the value of the Program. In the absence of comprehensive reporting of performance measurement, it was not possible to determine cost-effectiveness/value for money objectively.
2. There is general agreement among those interviewed that the Program should include a cost-recovery element. What is less clear is which elements of the Program should be covered by public funds and to what extent.
3. Some alternative delivery methods have been examined but have not been extensively explored.

4.4 Design and Delivery

This section will examine design and delivery -- the clarity of activity, accountabilities, expected deliverables, and intended results -- of the Environmental Emergencies Program. In addition, process considerations pertaining to the allocation of resources, management of risk, monitoring and reporting, and the leveraging of partnerships were considered.

• Overall, Program deliverers and senior managers consulted believe that current Program outputs and outcomes are reasonable and achievable but that outcomes need to be revisited to determine whether adjustments are necessary.
• Program deliverers reported that performance measurement of the Program's outcomes is still in the development phase; at present, only a database on spill data is in place (problems were identified with the database concerning the consistency of measurement of spill data). Major concerns were expressed by key informants regarding data collection and data integrity, performance measurement, and reporting.
• A document review revealed that outcomes for the Program are articulated in both the 2004 Results-based Management and Accountability Framework and the Outcome Project Plan (OPP). However, the outcomes vary significantly between these two documents.

• Outputs and outcomes for the Public Security and Anti-Terrorism component of the Environmental Emergencies Program are articulated in the Public Security and Anti-Terrorism annual reports (seven outcomes were identified in the 2003-2004 report and two were added in the 2005-2006 report).
• There is no structured performance measurement of these outcomes. Data are generated for the Public Security and Anti-Terrorism annual reports presented to Treasury Board but these reports provide information on activities and outputs only.

Findings:

Is the Program design consistent with its mandate? Are the activities and outputs of the Program consistent with its mandate and plausibly linked to the outcomes in terms of clarity and attribution?

• Program documents and key informant interviews demonstrate that Program activities are consistent with the mandate of the Program. However, although the logic model describes Program outcomes, outputs, and activities, the relationships between the activities of the Program and the outcomes are not clear. The format of the logic model does not allow for the individual activities and outputs to be linked to the immediate outcomes. It is unclear whether or not the outputs apply to one or more of the listed outcomes and whether all the outputs listed are achieved through one or more of the corresponding activities. Therefore, the Program's underpinning logic is not expressed in the logic model as clearly as it should be.

How consistent is the Program with its own proposed approach (has the Program been delivered as designed)?

• A majority of Program deliverers and senior managers interviewed indicated that there is a strong relationship between the legislation and the implementation of the Program; however, problems with consistency across the regions and available resources limit the effective implementation of the Program. Some respondents felt that more attention needs to be focussed on research and development and on prevention activities and outputs.

Are decision-making processes in place to allow for the highest areas of importance to be reflected in the allocation of resources (priorities)?

• Documentation revealed that the importance of setting priorities is emphasized and decision-making processes are outlined to some extent.
• Program management mentioned that priority-setting decisions are made at the Board level, resource allocation decisions are made at the Outcome Project Grouping (OPG) level, and more specific Program work plans are prepared at the Outcome Project Plan (OPP) level.

Is there an established structure that provides for responsive management and logically supports the achievement of goals and objectives? Who is accountable for the Program? Are the roles and responsibilities for all groups involved clear?

• A clear definition of current roles, responsibilities, and accountabilities of all groups involved in environmental emergencies could not be found in Program documents. Various documents such as the National Contingency Plan make reference to roles and responsibilities of stakeholders, but these documents are not up to date and therefore exclude some key components, such as the role of Public Safety Canada as well as the enhanced role of Environment Canada under the 2001 Public Security and Anti-Terrorism strategy.
• A majority of key informants across stakeholder groups consider the roles and responsibilities to be generally clear and well understood, although mixed responses were received from Environment Canada internal partners. Specifically, respondents from the Enforcement Branch and from Compliance Promotion units stated that roles and responsibilities are not clear or commonly understood. Most mentioned that roles and responsibilities regarding environmental emergency planning and management, enforcement, and compliance promotion activities related to the Canadian Environmental Protection Act, 1999 and the Fisheries Act are not formally documented at the program level, but are determined on an ad hoc basis or are based on existing relationships.

Findings:

• Based on a review of documentation, risk has been assessed by the Program at both the Outcome Project Grouping (OPG) and Outcome Project Plan (OPP) level. At the OPG level, two key risks (related to the Program) and their mitigation strategies were identified:
  • the transition to a sector-based risk management approach through Sector Sustainability Tables; and
  • the way in which the move of IM/IT capacity from programs to a centrally managed corporate function will affect the ability of IM/IT staff to understand the needs of the Environmental Emergencies Program and develop tailored Web tools.
• At the OPP level, a risk analysis was provided focusing on two external (stakeholders; political and reputation) and three internal (human resources capacity; financial resources; business activities and assets) risks, along with their likelihood and impacts. No mitigation strategy exists for these risks.
• Overall, no documentation was found to demonstrate that the various risk mitigation strategies have been implemented.
• Interview respondents identified one key risk, pertaining to the loss of human resources capacity, as many people are retiring. They identified two mitigation strategies for this risk: a national training plan and the "get big quick" program.¹¹

Findings:

• Some data were produced by the Program to allow its capacity requirements to be assessed. First, the Program examined planned vs. actual expenditures on salaries, showing that there is a shortfall in financial resources. Second, it calculated the number of staff (expressed in person-years) contributing to Program implementation in the Environmental Enforcement Division, the Environmental Science and Technology Centre, and each region. Finally, it conducted an assessment of the number of staff needed to deliver various aspects of the Program in each region, as well as HQ activities.
• Available documents enabled a comparison of needed vs. available human resources in the regions and HQ, showing that 50.9 staff (FTEs) are needed but only 37.8 are available in the regions, while 20.3 FTEs are needed but only 16.0 are available at HQ. However, there is no documentary evidence that specific capacity gaps were flagged or that data were used in the allocation of resources within the Program or to support a request for additional resources.
• According to key informants, the Program's capacity requirements are becoming an issue, particularly because of the retirement of key staff. Documentation shows that the budgets of the Environmental Emergencies Program have been affected by department-wide cuts and that expenditures exceed budgets. General information also shows a staff population nearing retirement and the need for succession planning.

Findings:

• Interview respondents generally agreed that the Program is consistent with the Public Security and Anti-Terrorism strategy, although some expressed concern that the environment is not a primary item within the security agenda.
• One of Environment Canada's three goals under the Public Security and Anti-Terrorism strategy is very similar to the mandate of the Environmental Emergencies Program: to implement measures to enhance the prevention of environmental emergencies and the preparedness, response, and recovery at facilities that manage materials that, if released accidentally or by deliberate action, would endanger human health or environmental quality.
• Activities reported by the Program since 2003-2004 indicate alignment with the intent of the Public Security and Anti-Terrorism funding allocation. In particular, the Environmental Emergencies Program contributed to the development of the Environmental Emergency Regulations and developed and maintains an inventory of substances considered toxic under the Canadian Environmental Protection Act, 1999.

Findings:

• Available documents provide only a breakdown of expenditures by activity between FY 2003-2004 and 2005-2006. No documentation is available to link these activities to specific Program outcomes. Performance indicators are identified for the outputs only, but since no performance data are provided it is difficult to determine the impact of the Public Security and Anti-Terrorism funding.
• Unable to determine whether outcomes are being achieved, as the three annual reports focus on activities/outputs and no additional documentation was available.
• In the annual reports for 2004-2005 and 2005-2006, key activities were repeated, along with additional activities in 2005-2006, including initiation of formal consultations with industry, environmental non-governmental organizations, provinces, territories, and other key stakeholders in July 2005, as well as a proposal to add 34 substances to the Environmental Emergency Regulations.
• Although Program deliverers were able to identify activities funded through additional Public Security and Anti-Terrorism resources, there is no performance information to demonstrate the actual results of these activities. Therefore, no objective evidence exists to show whether Public Security and Anti-Terrorism resources were appropriately used towards the achievement of intended outcomes.

Findings:

• Program deliverers interviewed strongly agree that the Program has been successful in establishing partnerships and leveraging outside resources in its research and development component.
• Anecdotal evidence demonstrates that ratios of leveraging from external sources run from 0.5:1 to as high as 5:1 (external/internal funding); however, no report on leveraging ratios exists. Senior management respondents agree that they do not have a solid understanding of how much leveraging takes place within the Program's research and development component. A document review revealed that actual ratios of leveraging have been declining since FY 2002-2003: 3.92 in FY 2002-2003; 3.64 in FY 2003-2004; 3.31 in FY 2004-2005; and 2.51 in FY 2005-2006.
• The Program research agenda is determined through five international committees that include in their membership representatives from the federal government, the regions, and U.S. federal government agencies. Through conferences and other avenues, the research agenda is also developed using input from other scientists, other government departments, and foreign agencies.
• Program documents show that several joint projects were implemented each fiscal year and that some were cost-shared with the project partner, based on signed agreements outlining the purpose and expected deliverables of the projects, responsibilities of both the contractor and the department, and their respective financial and in-kind contributions to the project.

Findings:

• The intended users of research and development outputs are industry, other government departments, academia, and other federal agencies. Evidence of the use of the Program's research outputs exists in the form of the number of hits received on its website (the majority of hits are directed at the Oil Properties Catalogue developed by the Program's research and development component). The use of other research outputs, such as scientific articles and conference proceedings, is not documented or monitored by the Program.
• Overall, more than half of interviewed partners from Environment Canada, other federal government departments, non-federal government departments, the regulated community, and research and development organizations reported having used the Program's research and development outputs. The stakeholder groups that make the most use of the Program's research and development outputs are the non-federal external partners and external non-governmental research and development organizations.
• Interview respondents indicated that they used outputs such as modelling and mapping systems, reports on decontamination methodologies, information from workshops, exercises and conferences, information on weather systems, publications, and manuals on spill response. The outputs are used for general information purposes as well as for specific activity areas such as preparedness, response, and recovery.
• There is insufficient documented evidence to determine how exactly stakeholders use research and development outputs (deliverables).

Findings:

a) Complementarity vs. duplication of the Environmental Emergencies Program and other programs

• Closely connected existing programs that address environmental emergencies are undertaken by provincial/territorial and municipal governments, Public Safety Canada, the Canadian Coast Guard, Fisheries and Oceans Canada, Transport Canada, and the Chemical, Biological, Radiological and Nuclear Research and Technology Initiative.
• A number of mechanisms have been put in place to reduce the likelihood of duplication and to ensure complementarity of the work of the various bodies involved in environmental emergency management and response in Canada. These include regional contingency plans that outline the roles and responsibilities of each stakeholder, Regional Environmental Emergencies Team (REET) planning meetings, MOUs and agreements between various stakeholders, and joint training and exercises. It must however be noted that no document could be found that clearly draws the boundaries of each area of responsibility, thereby leaving the possibility of some overlap.
• Overall, a majority of key informants consider that there is no duplication of the Environmental Emergencies Program by other programs. The work of the Environmental Emergencies Program and of the provinces and territories is seen as particularly complementary given that they have the same goals but their lines of jurisdiction are different.
• The areas of duplication, overlap, or unclear delineation of responsibility observed by a minority of respondents were as follows:
  • Environment Canada and Fisheries and Oceans Canada both address oil spills;
  • within Environment Canada both the Meteorological Service of Canada and the Environmental Science and Technology Centre conduct modelling;
  • some chemical industries' "responsible care" programs are similar to the Environmental Emergencies Program;
  • the provinces of Quebec and Ontario are contemplating conducting emergency planning work for their communities similar to the work already being carried out by the Environmental Emergencies Program;
  • the protection of wildlife falls under both provincial and federal jurisdiction; and
  • the responsibility for shoreline cleanup in British Columbia and for response level situations in Quebec is unclear.

b) Impacts if the Program no longer existed

• According to interview respondents, the likely impact on the emergency management system of Canada should the Environmental Emergencies Program be cancelled would be the emergence of gaps in the emergency management system in such areas as research capacity, scientific and technical expertise, central coordination and oversight, credibility of information, effectiveness of emergency response, communication to the public about hazardous materials, and ability to address interprovincial or international issues. These gaps could result in industry not knowing how to prevent incidents and would make it difficult for industry to understand the full spectrum of environmental emergency activities, leading to slower response rates to environmental emergencies and greater collateral damage.
• Various answers were obtained as to who could possibly fill these gaps. Most consider that the work of the Program could not be taken on by any other level of government; others thought that provinces and municipalities could work to fill some of these gaps but that they would need additional funding in order to do so; and a few thought that industry could assume this role.

Findings:

How successful has the Program been in working with partners?

• The majority of all interview respondents, both internal and external, believed that the Environmental Emergencies Program was working well with partners. The federal partners interviewed stated that the Program was successful, and 78% of the facilities surveyed indicated that the Program was meeting their needs to a moderate (53%) or a great extent (25%). The overall finding is that the Program is doing fairly well, but there remain certain areas for improvement outlined below.

What improvements to the Program can be made with respect to its work with partners?

• Respondents identified three improvements:
  
  • clarify and communicate roles and responsibilities;
  • increase communications to stakeholders (i.e., increase the number of training sessions and workshops); and
  • increase the level of resources, both personnel and financial (this would lead to greater leadership on the part of the Program).

What is the possibility of transferring Program activities to partners?

• Mixed responses were received from key informants relating to opportunities for transferring activities in whole or in part to a private/voluntary sector or to the provinces/municipalities. Some suggested possible options for transferring responsibilities to industry or provinces, while highlighting the challenges these transfers would entail, namely, that municipalities/provinces do not have the required resources and expertise to accept such a transfer. Others noted areas where overlap of responsibilities already occurs among industry, the provinces (Ontario, Quebec, and British Columbia have environmental emergency programs in place), and Environment Canada, while others stated that the Program should remain the responsibility of the federal government.

4.4.1 Summary Points for Design and Delivery

1. In those cases where outcomes are not being achieved, it is not clear, given the absence of an adequate performance measurement system, which specific activities are contributing to the achievement of the intended outcomes. This design and delivery flaw is also illustrated in the logic model. There is also the lack of integration into the Program's logic model and performance measurement strategy (specifically, into the prevention and preparedness elements of the Program) of activities and outcomes under the Public Security and Anti-Terrorism strategy.
2. There is a lack of a comprehensive and robust outreach strategy to ensure that the intended impact on client awareness is achieved, specifically, in the activity areas of prevention and preparedness.
3. The Program's awareness of who constitutes the regulated community is incomplete (i.e., what industry sectors are included, along with their relevant information).
4. A comprehensive procedure to verify information supplied by clients concerning compliance with the Environmental Emergency Regulations has not been implemented.
5. Although some elements of capacity analysis have been undertaken, there is no complete and comprehensive assessment of the Program's capacity.
6. The performance measurement system within the Program is inadequate. Specifically, there are issues with data collection and data integrity; a lack of identified indicators for outcomes under the Public Security and Anti-Terrorism strategy; and limited baseline data for outcomes.
7. The division of roles and responsibilities within Environment Canada with respect to compliance promotion, as well as enforcement, is not clear, well understood, or properly documented.

4.5 Stakeholder Opinions of Overall Strengths, Weaknesses, and Areas of Improvement

This section provides an overview of stakeholder opinions of overall strengths, weaknesses, and areas of improvement, stemming from key informant interviews and the survey of regulated facilities.

Findings:

• As a general unprompted comment, the interviews revealed five main strengths of the Program; two of the five were dominant. About 40% of interview respondents mentioned staff and personnel (in terms of science and technology expertise and support) as a key strength, and about 25% mentioned the Program's ability to form partnerships. The three other dominant characteristics identified were raising awareness (through education and training of stakeholders); scientific research and development capacity; and response to emergencies. The remaining comments were "one-offs."
• When asked specifically to identify the strengths and weaknesses of the Program, some 20% of respondents did not identify any specific weaknesses and 25% of respondents did not identify any specific strengths.
• There were three weaknesses that were dominant and four areas of improvement were specifically noted (8% or above) in response to a specific and prompted question. The weaknesses were a lack of communication with partners and stakeholders (12%), regional variability (internal unevenness) (10%), and lack of visibility and awareness (8%). In contrast, areas that should be improved were funding for financial and human resources (11%), public awareness/visibility (10%), human resources planning (mentoring and succession), and education and training (both at 8%).
• The survey of regulated facilities revealed the following:
  
  • While a variety of strengths were identified by survey respondents, two "results-based" outcomes emerged as the most common: raised awareness and education, and companies and people acting responsibly and being prepared for emergencies. Of note, the impact of the Program on raising awareness is mentioned more than twice as often as the impact of the Environmental Emergency Regulations on raising awareness.
  • The most frequently identified weakness, for both the Program and the Environmental Emergency Regulations, was lack of awareness and information. However, other factors related to awareness and information were also cited by smaller but sizeable percentages of respondents, including lack of enforcement of the regulations, auditing, and inspections; difficulty in understanding the regulations; lack of support or guidance; and duplication of services. Twenty-two percent of respondents could not identify any weakness of the Program, while this figure stands at 42% for the Environmental Emergency Regulations.
  • The survey results also reveal that increased awareness and additional information are the most commonly identified areas of improvement suggested by facilities representatives. Other notable improvement suggestions included increased involvement of other parties in the Program and increased enforcement. In general, the improvements that were suggested tend to be varied, and a large percentage of respondents did not identify any areas for improvement.

5.0 CONCLUSIONS

This is the first evaluation examining the Environmental Emergencies Program, one of the more mature program areas within Environment Canada. Generally, the evaluation of the Program is positive, as the Program is considered to be valuable and no fundamental problems were found. Presented below are the main conclusions of the evaluation, based on the findings presented in section 4.0.

1. The Environmental Emergencies Program continues to be relevant, as it does serve the public interest and there is a clear role for government; nevertheless, there is an opportunity to further redefine the precise role for each level of government while taking into consideration the federal government's legislated responsibilities.
2. The Environmental Emergencies Program is generally successful in meeting its intended outcomes. However, certain design and delivery elements limit quantitative measurement of the degree of success.
3. Although qualitative evidence suggests that the Environmental Emergencies Program is cost-effective, this finding could not be substantiated quantitatively.
4. There are a number of areas that need improvement in the design and delivery of the Environmental Emergencies Program. These include the implementation of an effective performance measurement system; a comprehensive assessment of the Program's capacity; updating the Program's logic model and performance measurement strategy by integrating the Public Security and Anti-Terrorism activities and outcomes into the prevention and preparedness elements of the Program; and the implementation of a comprehensive and robust outreach strategy.
5. Roles and responsibilities in the areas of compliance promotion and enforcement in the context of the Environmental Emergencies Program need to be clarified internally within Environment Canada.

6.0 RECOMMENDATIONS

Although areas of concern were raised in the evaluation with regard to the relevance, success, and cost-effectiveness of the Environmental Emergencies Program, these concerns are related to design and delivery aspects, and can therefore be addressed through adjustments in the design and delivery of the Program.

The Program continues to be relevant and Environment Canada should continue to carry out these functions; therefore, the Program continues to operate from a sound basis.

Given current pressures on the Program, known capacity issues (including succession), and ongoing financial pressures at the government level, it is urgent that response to the recommendations be undertaken. The recommendations¹² are presented in sequence, based on logical structure and the time frame in which they should be addressed.

Recommendation #1:

Presently, within the department there are specific groups with functional responsibility for enforcement and for compliance promotion activities. Simultaneously, the EEP has functional responsibility for these two activities in the context of environmental emergencies. This has led to some confusion internally as to where the functional responsibilities begin and end for these two activities. This concern was raised by several internal and external key informant interview groups.

The evaluation also found that a number of areas need to be addressed with respect to the design and delivery of the Program:

• in the areas where Program outcomes are not being achieved, it is not clear, given the absence of an adequate performance measurement system, which specific activities are contributing to the achievement of the intended outcomes (this also applies to the logic model);
• a comprehensive assessment of the Program's capacity needs to be carried out;
• the Program's impact on client awareness needs to be comprehensive;
• a comprehensive procedure to verify information supplied by clients regarding compliance with the Environmental Emergency Regulations needs to be implemented; and
• the Program's logic model and performance measurement strategy need to be updated by integrating the Public Security and Anti-Terrorism activities and outcomes into the prevention and preparedness activity areas of the Program.

The EP Board should undertake to address the following recommendations in the order presented:

1. in collaboration with the Chief Enforcement Officer, clarify and adjust, where necessary, roles and responsibilities in the context of the Program, with a particular focus on compliance promotion and enforcement.
2. conduct further research regarding the activities related to raising awareness in the prevention and preparedness areas of the Program to determine which of these activities are not contributing to the achievement of these outcomes.
3. conduct a comprehensive program capacity assessment (i.e., needed vs. actual); where gaps emerge determine risks and implement mitigation strategies.
4. finalize and implement a performance measurement and reporting system.
5. address the Program's monitoring gaps (i.e., verification; client knowledge and outreach strategies).

Previously, environmental emergency planning and management, enforcement, and compliance promotion activities related to the Canadian Environmental Protection Act, 1999 and the Fisheries Act were located under the same Outcome Project Grouping (OPG) under the Environmental Protection (EP) Board. Thus, there was regular communication among the Environmental Emergencies Program, Enforcement, and Compliance Promotion and awareness of each other's activities. A decision was made in April 2006 to consolidate all enforcement activities within the department under the Departmental Management Services (DMS) Board. Presently, the fact that these activities are no longer consolidated under a single governance structure has exacerbated the confusion as to where functional responsibilities begin and end for these activities.

The activities and expected outcomes of the Environmental Emergencies Program need to be aligned with clearly defined roles and responsibilities. This may necessitate a revision of the Program's logic model. This would also provide an opportunity to update the Program's logic model and performance measurement strategy by integrating the Public Security and Anti-Terrorism activities and outcomes into the prevention and preparedness activity areas of the Program. The mandate of the evaluation was to examine the four evaluation issues (relevance, success, cost-effectiveness, and design and delivery) across the five activity areas of the Program. In those areas of the Program where outcomes were not fully met (e.g., awareness), the evaluation was not able to examine in greater depth which specific activities were or were not successful, because no adequate performance measurement system for the Program has been implemented. Building upon the findings of this evaluation, therefore, further research is necessary in these areas to identify which specific activities contributed to the achievement of these outcomes, with the objective of identifying areas to be improved and thereby enhancing the Program's efficiency and effectiveness. Part of this research may necessitate the collection of data concerning specific activities in question to determine to what extent these activities were successful. Once the Program has identified which awareness activities have been successful, the Program should then develop and implement a comprehensive outreach strategy.

A comprehensive assessment of the Program's capacity needs to be conducted to identify gaps, if any, between available and needed capacity. If gaps are identified, related risks should be determined and mitigation strategies should be implemented to address these risks.

The Program should finalize and implement a performance measurement and reporting system, identifying key indicators and the data necessary to measure the Program's success in achieving its stated objectives.

The Program has specific monitoring gaps in the area of compliance that need to be examined and addressed to ensure proper linkages with the population on which it principally has an impact.

7.0 MANAGEMENT RESPONSE

Recommendation 1a:

The EP Board should, in collaboration with the Chief Enforcement Officer, clarify and adjust, where necessary, roles and responsibilities in the context of the Program, with a particular focus on compliance promotion and enforcement.

Management Response:

EP Board agrees with this recommendation. Enforcement, regulatory compliance promotion and the provision of scientific and technical information and advice are three distinct departmental functions within Environment Canada. For legal as well as administrative reasons it is imperative that staff roles and responsibilities are clearly defined and understood both internally within the departments as well as by external stakeholders with whom such staff interact.

EP Board's delegated representative and the Chief Enforcement Officer have agreed to work towards development, by March 2008), of a governance document to clarify the relationship between emergency, compliance promotion and enforcement activities. This document will be developed according to Quality Management System (QMS) methodology.

Recommendation 1b:

The EP Board should conduct further research regarding the activities related to raising awareness in the prevention and preparedness areas of the Program to determine which of these activities are not contributing to the achievement of these outcomes.

Management Response:

EP Board agrees with this recommendation. With the exception of compliance promotion for the Environmental Emergency Regulation under Part 8 of CEPA 1999, the Department's historical approach to raising the awareness of our external stakeholders has, to a large extent, been to take advantage of regional and local initiatives led by other governments and OFGDs or private sector stakeholders. This approach has resulted in varying degrees of success.

EP Board commits to carry out a study that will identify key target audiences and assess the strengths, weaknesses and costs associated with a number of delivery mechanism options for raising emergency prevention and preparedness awareness. This study will be completed by June, 2008.

Recommendation 1c:

The EP Board should conduct a comprehensive program capacity assessment (i.e., needed vs. actual); where gaps emerge determine risks and implement mitigation strategies.

Management Response:

EP Board agrees with this recommendation. Significant work has been done in the past in determining resource needs required to effectively deliver the mandated responsibilities of the Environmental Emergencies Program. EP Board will draw upon this previous work to produce a comprehensive program wide assessment of capacity requirements, gaps, risks and mitigation strategies in relationship to Environment Canada's mandated roles and responsibilities. This assessment will also factor in the results of the awareness study which is to be undertaken in response to Recommendations #2 above and set forth options for management consideration. This work will be completed by January, 2009 so as to be ready in time for the FY 09/10 planning process.

Recommendation 1d:

The EP Board should finalize and implement a performance measurement and reporting system.

Management Response:

EP Board agrees with this recommendation. Significant work has been carried out over the past several years in identifying a comprehensive set of performance indicators for the prevention, preparedness, response, recovery and R&D components of the Environmental Emergencies Program. Effort must now be directed towards analysing and selecting a smaller representative core set of indicators that will be effective in measuring program outcomes and effectiveness and which can be implemented in a nationally consistent and affordable manner. This work will be completed in parallel with the comprehensive program capacity assessment referenced above in response to Recommendation #3 and as such will be completed by January, 2009.

Recommendation 1e:

The EP Board should address the Program's monitoring gaps (i.e., verification; client knowledge and outreach strategies).

Management Response:

EP Board agrees with this recommendation and considers the development of an effective as well as affordable performance measurement strategy, as referenced above in response to Recommendation #4, as being critical to addressing the concerns identified in the evaluation of the Program.

Annex 1
Evaluation Issues and Questions

Annex 2
List of Background and Supporting Documentation

Annex 3
List of Interviewees and Interview Guides

Master List -- Interview Questions

Introduction:

The Audit and Evaluation Branch of Environment Canada is conducting an evaluation of the Environmental Emergencies Program (EEP). The purpose of this evaluation is to examine whether the Program is consistent with organizational priorities and addressing an actual societal need; achieving its intended outcomes; using the most appropriate and efficient means to achieve the outcomes; and being delivered in the best possible way. All the activity areas of the EEP (prevention, preparedness, response, recovery and research and development) are included in the evaluation.

In accordance with best practices, the approach for the evaluation involves the use of multiple lines of evidence and complementary research methods. To this end, interviews with selected key informants have been planned. The questions below serve to guide this interview process.

Overview Questions:

1. Please describe your role and experience with the EEP. Which areas of the EEP are you most familiar or involved with? Note: Areas include prevention, preparedness, response, recovery and research and development.

Partners and stakeholders only
2. Please describe the working relation between the EEP and your organization.

Specific Questions:

3. For the most part, when government programs are designed it is usually with the intent to address concerns and priorities of society.

In your opinion, to what degree is the EEP addressing societal needs? Please describe. [REF: 1a]

4. To what degree does the EEP contribute to the CESF outcome? [Attain the highest level of environmental quality as a means to enhance the well-being of Canadians, preserve our natural environment, and advance our long-term competitiveness]? Please describe. [REF: 1b]

5. To what degree does the EEP contribute to delivering departmental outcomes and Board priorities? Please describe. [REF: 1c]

6. To what degree is the EEP aligned with the Public Safety and Anti-Terrorism objectives [Namely, improving air security, enhancing screening at borders, heightening border security and facilitation, increasing effective intelligence and policing, enhancing emergency preparations and support for the military and building on border infrastructure and international capacity]? Please describe. [REF: 1d]

7. Historically, the government has played a role with regards to environmental emergencies.

To what extent do you feel that a government role in environmental emergencies is necessary? Please describe. [REF: 2]

8. In your opinion, do you feel that federal government funding and activities in environmental emergencies should continue or should part or the entirety be realigned to the provinces and territories or to other programs? Please elaborate. [REF: 3]

9. How does the EEP take into consideration the diverse needs and circumstances of provinces, territories and regions within the context of a coordinated national program? [REF: 3]

10. Are there any programs [including non-federal government programs] that have similar objectives and activities to the EEP? [Prompt: Are there any other programs that duplicate or complement the objectives and activities of the EEP?] [REF: 4a]

1. If so, please describe the aspects of the programs that are similar and the ones that are complementary.

11. What gaps would emerge in the emergency management system of Canada if the EEP did not exist? [REF: 4b]

1. What would be the impacts of these gaps?
2. Would/could others step in to fill in these gaps?

12. A Results-Based Management and Accountability Framework (RMAF) was developed in 2004 for the EEP. This RMAF contained logic models outlining the specific outcomes to be achieved by the EEP.

To what degree are the EEP outcomes (immediate, intermediate and long-term) reasonable given the scope of EEP activities? [REF: 5a(i)]

1. Do the EEP outcomes need to be revised?

13. To what degree are the EEP outcomes commonly understood and accepted by all parties engaged in the EEP? [REF: 5a(i)]

14. What performance measurement and reporting strategies are in place to measure the achievement of the EEP outcomes? [REF: 5a(ii)]

1. What data has been captured thus far?
2. How are these data used?

15. To what degree have the following EEP immediate outcomes been achieved? Please describe. [Prompt: Could you please provide examples and/or supporting documentation?] [REF: 5a1a, 5a1b, 5a3a, 5a3b, 5a5a, 5a5b, 5a7a, 5a9a]

1. Increased awareness by targeted stakeholders on the nature and scope of risks for environmental emergencies, and appropriate mitigation measures (Prevention)
2. Clear and concise compliance and enforcement strategies to be applied to s.200 of CEPA 1999 [and the Fisheries Act] (Prevention)
3. Departmental readiness to provide timely, relevant scientific and technical advice and support consistent with mandated responsibilities (Preparedness)
4. Improved client awareness and readiness to manage environmental emergencies (Preparedness)
5. Coordinated timely, relevant scientific and technical (S&T) advice and support related to the department's environmental emergencies mandate (Response)
6. Environmental emergencies, where EC has the lead, are managed successfully (Response)
7. Increased awareness by targeted stakeholders on the assessment and restoration of damages caused by environmental emergencies (Recovery)
8. Advancement [development] of scientific knowledge, technologies, tools and approaches associated with environmental emergency prevention, preparedness, response and recovery (Research and Development)

16. To what degree have the following EEP intermediate outcomes been achieved? Please describe. [Prompt: Could you please provide examples and/or supporting documentation?] [REF: 5a2a, 5a4a, 5a6a, 5a8a, 5a10a]

1. Client implementation of emergency prevention, response and recovery plans and practices
2. Increased [R&D] investment in environmental emergency prevention and preparedness by industry and governments

17. To what degree has the EEP long-term outcome been achieved? Please describe. [Prompt: Could you please provide examples and/or supporting documentation?] [REF: 5a11]

1. Reduction in the frequency, severity and consequences of environmental emergencies that affect Canada

18. In 2001, the EEP's responsibilities increased due to the development of the Government of Canada's Public Security and Anti-Terrorism (PSAT) Strategy, which provided funds to enhance and implement security measures to protect the health and safety of Canadians. The EEP identified outcomes related to the Environmental Emergency Regulations, documented in the annual PSAT reports to Treasury Board.

To what degree are EEP PSAT outcomes (immediate and intermediate) reasonable given the scope of the EEP and the federal PSAT Strategy objectives? [REF: 5b(i)]

1. Do the EEP PSAT outcomes need to be revised?

Note: The PSAT strategy objectives are the following:

• improving air security;
• enhancing screening at borders;
• heightening border security and facilitation;
• increasing effective intelligence and policing;
• enhancing emergency preparations and support for the military; and
• building on border infrastructure and international capacity

19. What performance measurement and reporting strategies are in place to measure the achievement of the EEP PSAT outcomes? [REF: 5b(ii)]

1. What data has been captured thus far?
2. How are these data used?

20. To what degree have the following EEP PSAT immediate outcomes been achieved? Please describe. [Prompt: Could you please provide examples and/or supporting documentation?] [REF: 5b1a, 5b1b, 5b1c, 5b1d, 5b1e, 5b1f]

1. Increased awareness of the E2 Regulations
2. Increased EC's awareness of the regulated community
3. Increased community awareness of the E2 Regulations and of surrounding risks
4. Increased community participation in environmental emergency plan preparation
5. Increased partnerships/relationships between government and industry
6. Increased compliance with the E2 Regulations

21. To what degree has the EEP PSAT intermediate outcome been achieved? Please describe. [Prompt: Could you please provide examples and/or supporting documentation?] [REF: 5b2]

1. Improved E2 management at facilities and communities including product substitution to less hazardous substances -- prevention, preparedness, response and recovery

22. What external factors outside the EEP, either positive or negative, have facilitated or detracted from the achievement of the outcomes of the EEP? [REF: 6]

1. What have been the impacts of the external factors?

Program senior management
   a.  How were these factors addressed?

23. Has the EEP produced any unanticipated results, either positive or negative? [Prompt: Have there been any results, either positive or negative, that were unexpected?] [REF: 7]

1. If so, what were these unanticipated results and how were they addressed?

24. It is important that government programs be efficient and make a contribution to the public good.

In your opinion, to what degree are Canadians getting value for their tax dollars from the EEP? [Prompt: Included in the concept of the value for tax dollars are the aspects of relevance, impacts and costs.] [REF: 8]

Program deliverers and senior management
25. In your opinion, to what degree is the EEP using the most effective approaches to achieve its intended results? [REF: 10]

1. Would there be alternative delivery approaches that would make the EEP more effective? Please describe.

26. What alternative approaches, if any, have been examined to better achieve the intended results of the EEP? [REF: 10]

Partners
27. In your opinion, to what degree is the EEP using the most effective approaches to achieve its intended results? [REF: 10]

1. Would there be alternative delivery approaches that would make the EEP more effective? Please describe.

28. What activities were financed by the additional resources that the EEP received from the 2001 "security" budget? [REF: 11]

29. To what degree has the EEP explored:

1. partnerships in R&D?
2. external funding opportunities in R&D? [Prompt: What types of approaches have been used to explore partnerships and external funding opportunities?] [REF: 12]

30. In your opinion, how successful has the EEP been in leveraging outside resources? [Prompt: How successful has the EEP been in establishing partnerships and/or receiving funding for R & D?] [REF: 12]

Program deliverers and partners (internal & external)
31. In your opinion, to what extent should the EEP set up user fees to cover some of the costs incurred in providing its services? [REF: 13]

32. Are there any fee structures in place to cover some of the costs of the EEP? [REF: 13]

1. If yes, what are their objectives? [Prompt: What costs are covered?]
2. How successful have they been?

Regulated community and federal departments
33. Are there any specific services provided by the EEP for which you pay? [REF: 13]

1. If yes, how do you feel about paying for these services?

34. The mandate and outcomes of the EEP relate to the reduction of the frequency, severity and consequences of environmental emergencies by promoting prevention of and preparedness for environmental emergencies, providing response and recovery advice and advancing emergency science and technology.

Do you think the activities and outputs of the EEP contribute to this mandate? Please describe. [REF: 14]

1. Do you believe that there are any activities and/or outputs that are not needed?
2. Are there any activities and/or outputs missing?

35. The mandate of the EEP was derived from an assortment of federal legislation and policies, such as the 1973 Cabinet Directive establishing the roles and responsibilities of EC in relation to emergencies, the 1985 Emergency Preparedness Act and the 1999 Canadian Environmental Protection Act.

To what extent has the EEP been implemented as designed by these policies and legislation? [REF: 15]

36. How are decisions made in regards to:

1. the determination of priorities?
2. the allocation of resources? [Prompt: Please describe and provide supporting documentation.] [REF: 16]

37. What types of approaches have been used to identify and manage risk associated with the achievement of the EEP outcomes? Please describe. [Prompt: Examples of risks include financial, operational, political, technological, health and safety and environmental. Have relevant strategies been developed?] [REF: 17]

38. In your opinion, to what extent are the currently allocated human and financial resources adequate for the achievement of the outcomes of the EEP? Please describe. [REF: 18]

39. Has an analysis of capacity requirements (human and financial) been completed for the EEP? [REF: 18]

1. If yes, how are the results of the analysis used to allocate resources within the EEP?
2. If yes, have proposals for increased capacity been completed? [REF: 18]
   1. If yes, how is this analysis of capacity requirements included in the proposal for increased capacity?

40. The CESF was created in 2004 with the vision to attain the highest level of environmental quality as a means to improve Canadians' quality of life. It is supported by five pillars; decision-making, information, science and technology, performance promotion and enforcement, and education and engagement.

In your opinion, to what extent are the CESF pillars taken into consideration with regards to the current EEP structure? [REF: 19]

Program deliverers and senior management
41. What types of approaches have been used to reach potential partners? [REF: 20]

1. In your opinion, to what degree has the EEP been successful in working with relevant partners and stakeholders? [Prompt: What structures have been created?]
   1. Please describe the nature of these partnerships.
2. To what degree have the partners and stakeholders been engaged?
   1. Please describe the working relation between the EEP and the partners and stakeholders.

42. In your opinion, what opportunities, if any, are there to transfer some of the activities or components of the EEP to the private/voluntary sector or to the provinces/municipalities? [Prompt: Are there any other programs that duplicate or complement the objectives and activities of the EEP?] [REF: 20]

Partners and stakeholders
43. In your opinion, to what degree has the EEP been successful in working with your organization? [REF: 20]

1. What works well?
2. What could be improved?

44. In your opinion, what opportunities, if any, are there to transfer some of the activities or components of the EEP to the private/voluntary sector or to the provinces/municipalities? [Prompt: Are there any other programs that duplicate or complement the objectives and activities of the EEP?] [REF: 20]

Program deliverers and senior management
45. How were your roles and responsibilities related to the EEP established? [REF: 21]

46. In your opinion, to what degree are the roles and responsibilities of all parties involved in the EEP clear and commonly understood? Please elaborate. [REF: 21]

47. To what degree are the accountability relationships within the EEP clear and commonly understood? Please elaborate. [REF: 21]

1. To whom are you accountable?

Partners
48. How were your roles and responsibilities related to the EEP established? [REF: 21]

49. In your opinion, to what degree are the roles and responsibilities of all parties involved in the EEP clear and commonly understood? Please elaborate. [REF: 21]

All except program deliverers R&D
50. To what extent have you used the outputs/deliverables developed by the EEP R&D activity area? [REF: 22]

1. What was the purpose of the outputs/deliverables used?

Program deliverers R&D only
51. How are the outputs/deliverables developed by the EEP R&D activity area used by stakeholders, including the other activity area of the EEP? [REF: 22]

Concluding questions:

52. In your opinion, what are the key strengths and weaknesses of the EEP?

1. Do you have any suggestions to improve the EEP?

53. Do you have anything additional to add?

Annex 4
Summary List of Ratings by Question

Below is a table showing the individual summary ratings for the evaluation issues based on judgement of whether the findings indicate no problem (√), no problem, but based solely on subjective evidence (~√), a small problem (º), a major problem (×), or a rating was not applicable.

Out of the 39 issues originally identified in the evaluation framework,¹⁴ the evaluation of the Environmental Emergencies Program produced the following findings:

• 15 questions were determined to represent no problem (in the case of 4 of these, compelling subjective evidence was available but lack of performance data prevented a complete assessment);
• 16 were assessed as constituting a small problem;
• 3 were considered to represent a major problem;
• 2 of the issues were deemed not applicable (N/A); and
• 2 issues could not be assessed because performance data considered essential for assessment were not available.

¹ Environment Canada's Audit and Evaluation Plan for 2005-2006 was approved by the Departmental Audit and Evaluation Committee (DAEC) on June 15, 2005.

² Under the Canadian Environmental Protection Act, 1999, a "substance" includes any distinguishable kind of organic or inorganic matter, whether animate or inanimate, that is capable of being released as a single substance, an effluent, emission, waste, or a mixture into the Canadian environment.

³ The 1973 Cabinet directive is a Cabinet decision defining the roles and responsibilities of the Department of the Environment during emergencies.

⁴ These outcomes are not included in the 2004 Results-based Management and Accountability Framework.

⁵ Funding received for research and development projects from external parties (i.e., Chemical, Biological, Radiological and Nuclear Research and Technology Initiative, other government departments).

⁶ Not calculated in FY 2003-2004 budget; actual allocation of $2 million in CEPA Operational Review funding to regions and to the Emergencies Science and Technology Division (ESTD) not available for FY 2003-2004; for FY 2004-2005 and 2005-2006, CEPA Operational Review funding is included in budgeted amounts.

⁷ Outcomes were extracted from the Environmental Emergencies Program Results-based Management and Accountability Framework; outcomes related to the Public Security and Anti-Terrorism strategy were taken from the PSAT annual reports.

⁸ NOTE: Rating is a judgment of whether the findings indicate no problem (√), no problem, but based solely on subjective evidence (~√), a small problem (º), or a major problem (×). Annex 4 presents a summary list of these ratings for all the evaluation questions.

⁹ It is important to note that the polluter is responsible for costs whether or not the spill is accidental; the polluter pays principle does not infer liability but rather responsibility.

¹⁰ It is important to note that the Environmental Damages Fund does not provide for any such cost recovery.

¹¹ To deal with under-capacity in some areas, the number of environmental officers is supplemented from other areas.

¹² The purpose of the panel of experts was to review the draft evaluation report with a particular focus on the conclusions and recommendations. Overall, the panel was in agreement with the conclusions and recommendations of this report.

¹³ Treasury Board Secretariat’s Integrated Risk Management Framework defines risk as follows: Risk refers to the uncertainty that surrounds future events and outcomes. It is the expression of the likelihood and impact of an event with the potential to influence the achievement of an organization's objectives.

¹⁴ One of the original questions in the evaluation framework (Question 19 in Annex 1) dealing with the issue of the Competitiveness and Environmental Sustainability Framework (CESF) pillars was dropped.