Evaluation of the Canadian Environmental Sustainability Indicators (CESI) Initiative

Final Report

February 26, 2009

Table of Contents

• EXECUTIVE SUMMARY
• Management Response
• 1.0 INTRODUCTION
• 2.0 BACKGROUND
  • 2.1 Origins of the CESI Initiative
  • 2.2 Description of the Initiative
• 3.0 METHODOLOGY
  • 3.1 Approach
  • 3.2 Evaluation Issues
  • 3.3 Limitations of the Evaluation
• 4.0 FINDINGS
• 5.0 CONCLUSIONS
• 6.0 RECOMMENDATIONS
• 7.0 MANAGEMENT RESPONSE
• Annex 1 CESI Logic Model
• Annex 2 Key Secondary Sources Reviewed
• Annex 3 Complete List of Secondary Sources
• Annex 4 Key Informant Interview Guide
• Annex 5 Summary of Findings

Report Clearance Steps

Planning phase completed	June 2008
Report sent for management response	January 2009
Management response received	January 2009
Report completed	January 2009
Report approved by Departmental Evaluation Committee (DEC)	February 2009

Acronyms used in the report

CAPMoN	Canadian Air and Precipitation Monitoring Network
CCME	Canadian Council of Ministers of the Environment
CEPA	Canadian Environmental Protection Act
CESI	Canadian Environmental Sustainability Indicators
CISE	Canadian Information System for the Environment
DG	Director General
ENGO	Environmental non-governmental organization
ESDI	Environmental and Sustainable Development Indicators
FSDA	Federal Sustainability Development Act
FSDS	Federal Sustainability Development Strategy
GDP	Gross domestic product
IISD	International Institute for Sustainable Development
IPCC	Intergovernmental Panel on Climate Change
IS	Internal Services
NAPS	National Air Pollution Surveillance
NGO	Non-governmental organization
NPRI	National Pollutants Release Inventory
NRTEE	National Round Table on the Environment and the Economy
OECD	Organization for Economic Co-operation and Development
PM2.5	Fine particulate matter
SIID	Strategic Information Integration Directorate
WQI	Water Quality Index

Acknowledgments

The Evaluation Project Team, including Karine Kisilenko and Martine Perrault under the direction of the Director, Shelley Borys, would like to thank those individuals who contributed to this project and particularly all interviewees who provided insights and comments crucial to this evaluation.

Prepared by the Audit and Evaluation Branch of Environment Canada.

EXECUTIVE SUMMARY

Introduction

This report presents the results of the evaluation of the Canadian Environmental Sustainability Indicators (CESI) initiative conducted by Environment Canada’s Audit and Evaluation Branch between March and December 2008.

This evaluation focused on assessing the CESI initiative’s relevance, success, cost effectiveness and design and delivery since its inception in 2005. It relied on a review of documents, including recent external st udies of the initiative’s relevance and effectiveness, and 11 key informant interviews with CESI managers from Environment Canada, Statistics Canada and Health Canada.

The CESI initiative is a collaborative effort of Environment Canada, Statistics Canada and Health Canada, with input from Canadian provinces and territories initiated by the Government of Canada to develop and report on a small set of priority environmental indicators, notably air quality, water quality and greenhouse gas emissions. The indicators are intended to help provide Canadians with a better understanding of the relationships that exist among the economy, the environment and human health and well-being with respect to air quality, water quality and greenhouse gas emissions. The indicators are also intended to assist those in government who are responsible for developing policy and measuring performance.

The funding for the initiative includes $45 million for fiscal years 2005–2006 to 2008–2009, divided among Environment Canada ($32.5 million), Statistics Canada ($10.5 million) and Health Canada ($2 million).

Summary of Findings and Conclusions

Overall, evidence shows that the CESI initiative is relevant to both federal government priorities and various stakeholders’ environmental information needs. While other similar sources of environmental sustainability data exist, the initiative appears to have a unique mandate and to involve collaboration among the key government stakeholders (i.e., Environment Canada, Health Canada, Statistics Canada, and the provinces and territories).

However, while the CESI initiative has produced its intended outputs and achieved or is on track to achieving most of its outcomes pertaining to increased capacity to produce environmental indicators, no evidence was found that CESI products are actually being used, except by Treasury Board in its annual Canada’s Performance reports and by Human Resources and Social Development Canada in its set of human well-being indicators. Attention is needed with respect to achieving intermediate outcomes targeting the use of CESI information by decision makers and the general public, as well as the integration of CESI results in policy decisions.

In 2007, CESI managers contracted two external studies aiming, among other objectives, to measure the awareness and perceived usefulness of CESI products by CESI target users, namely, senior level policy makers, experts in accountability reporting, members of the general public with an interest in environmental issues, and representatives from environmental non-governmental organizations (ENGOs) and industry. Results from these studies revealed that CESI products were not well known and had design flaws that limited their usefulness.

In order to address these shortcomings, and based on recommendations from these two studies, CESI managers are currently making enhancements to the CESI website and the 2008 suite of products. These improvements include presenting CESI information in more common language, giving access to data at more specific levels (drill-down capacity), providing more trend information, and better integrating socio-economic and health context information.

One recommended change could, however, not be implemented. Several federal government policy makers and a few CESI representatives argued that the use of CESI data by federal government decision makers and policy makers would increase if it could be used for policy development and accountability reporting purposes. This is, however, seen as being dependent on the development of a federal environmental sustainability strategy and the identification of specific objectives against which the federal government would be measured using CESI indicators. No such objectives currently exist.

In light of the improvements that are currently being made to the initiative, it is too early to draw definitive conclusions on the adequacy of the CESI initiative to meet its target users’ need. Once these changes are fully implemented, the initiative will need to demonstrate its usefulness by documenting who its target users are and by monitoring the extent to which CESI products are known, accessed, used and considered useful by these target users.

The initiative currently lacks a performance measurement system to effectively monitor whether its target users are accessing and using the CESI information. The number of responses to the two user surveys conducted by Statistics Canada was too low for the results to be meaningful. Furthermore, evidence shows that there is generally low awareness of CESI products among the general public and non-government stakeholders, suggesting that a more effective information dissemination strategy may also be required.

Nevertheless, the initiative is generally implemented as designed and in an efficient manner. It benefits from an appropriate and effective governance structure that reflects the interdepartmental nature of its mandate and activities. CESI managers have also demonstrated a best results-management practice in producing meeting minutes, records of decisions and a review of CESI implementation challenges that led to effective solutions being developed.

Recommendations

The following recommendations are based on the findings and conclusions of the evaluation.

Recommendation 1: In order to demonstrate achievement of its intermediate outcomes pertaining to the use of CESI products by its target audiences, the initiative needs to monitor the extent to which CESI products are known, accessed, used and considered useful by its target users, and commit to regular reporting on the reach and use of the CESI products. It is recommended that the CESI DG Steering Committee oversee development and implementation of a performance measurement system for the CESI initiative. This system should include a database of target users of CESI products, a process for annually measuring user reach, satisfaction and use of CESI products, and a reporting strategy.

Recommendation 2: Several sources indicated that the target CESI users have limited awareness of the CESI products and website. Given the importance of increasing awareness of the CESI products, it is recommended that the CESI DG Steering Committee revisit the CESI communications strategy to identify and implement means to increase target CESI users’ awareness of CESI products.

Recommendation 3: In light of the significant resource investments made in the CESI initiative, it is important for the implicated federal departments to maximize the use and usefulness of the CESI products. All sources consulted for this evaluation indicate that the current improvements being made to the 2008 CESI products suite are likely to improve the usefulness of CESI products. Some sources, however, consider that the use of CESI for policy development and government decision making is hindered by the absence of a federal government commitment to specific and measurable objectives related to the CESI indicators.

a) It is recommended that once intended CESI improvements are completed, the CESI DG Steering Committee reassess the success of the CESI initiative.

b) It is recommended that the DG Steering Committee consider the role expected of CESI in supporting policy development and federal government decision making, and explore additional means to increase CESI’s usefulness such as possible linkages to the implementation of the 2008 Federal Sustainable Development Act.

Management Response

The CESI initiative, on behalf of the Government of Canada, demonstrates Canada’s ability to tell the country’s sustainability story and report to Canadians on the state of the Canadian environment. For the past four years, CESI has demonstrated a record of organizing and disseminating information related to the environment in a scientifically credible and defensible way and tracking and reporting trends on air quality, water and greenhouse gas emissions.

Funding for the CESI initiative was initially approved in 2003–2004 for a period of five years. Since the funding sunsets in 2008–2009, Environment Canada, in partnership with Statistics Canada and Health Canada, is seeking opportunities for the renewal of the CESI initiative. All Management Response commitments to the Evaluation Recommendations will be dependent on the nature of the renewal of the initiative and the related available resources.

The Assistant Deputy Minister of the Strategic Policy Branch and the CESI DG Steering Committee Chair, Director General, Strategic Information Integration Directorate, accept the evaluation and its recommendations and have provided a preliminary plan to implement the recommendations within the context of a possible CESI renewal.

Recommendation 1: In order to demonstrate achievement of its intermediate outcomes pertaining to the use of CESI products by its target audiences, the initiative needs to monitor the extent to which CESI products are known, accessed, used and considered useful by its target users, and commit to regular reporting on the reach and use of the CESI products. It is recommended that the CESI DG Steering Committee oversee development and implementation of a performance measurement system for the CESI initiative. This system should include a database of target users of CESI products, a process for annually measuring user reach, satisfaction and use of CESI products, and a reporting strategy.

Response: The CESI DG Steering Committee agrees with Recommendation 1 to develop and implement a performance measurement system for the CESI initiative. In response, within three months of the renewal of CESI, the CESI DG Steering Committee will develop and present to the Internal Services (IS) Board, for its approval, a revised CESI Indicators Logic Model and a Performance Measurement Plan that will include clear accountabilities and timelines for implementation of the performance measurement plan and associated data collections mechanisms.

Performance measurement will include:

• An inventory of target users of CESI products
• A process for annually measuring user reach, satisfaction and use of CESI products, (monitoring client feedback on CESI products through focus groups/public opinion research and through web monitoring of client statistics which will provide valuable information to ensure that the CESI products meet federal government policy development and decision-making needs) and
• A reporting strategy

Recommendation 2: Several sources indicated that the target CESI users have limited awareness of the CESI products and website. Given the importance of increasing awareness of the CESI products, it is recommended that the CESI DG Steering Committee revisit the CESI communications strategy to oversee, identify and implement means to increase target CESI users’ awareness of CESI products.

Response: The CESI DG Steering Committee agrees with the recommendation to develop a more strategic and targeted marketing strategy. Within three months of the renewal of CESI, the CESI DG Steering Committee will present to the IS Board a CESI Engagement Strategy that will involve collaboration with the Department’s Communications Branch to develop and implement a strategy to identify the intended target audiences and promote awareness and use of the CESI report by Fall 2009.

This strategy would demonstrate the Government of Canada’s commitment to accountability and transparency to inform Canadians in an open and transparent manner about key environmental issues. The new Engagement Strategy will include: key principles and considerations, results of audience research, outreach objectives and proposed activities, monitoring and evaluation, budget and plan for implementation and campaign management. The strategy will address, in particular, engagement of interested partners to champion the dissemination and promotion of the CESI products.

Recommendation 3: In light of the significant resource investments made in the CESI initiative, it is important for the implicated federal departments to maximize the use and usefulness of the CESI products. All sources consulted for this evaluation indicate that the current improvements being made to the 2008 CESI products suite are likely to improve the usefulness of CESI products. Some sources, however, consider that the use of CESI for policy development and government decision making is hindered by the absence of a federal government commitment to specific and measurable objectives related to the CESI indicators.

a) It is recommended that once intended CESI improvements are completed, the CESI DG Steering Committee reassess the success of the CESI initiative.

Response: The CESI DG Steering Committee agrees with the recommendation to reassess the success of the CESI initiative. The CESI DG Steering Committee will initiate the intended improvements in the context of the renewal of CESI and will implement a cycle of evaluation of the initiative, aligned with the renewal time frame (i.e., mid term and final evaluation in a 5-year cycle). The CESI Indicators Logic Model and Performance Measurement Plan, developed in response to Recommendation 1, will be the basis for monitoring and tracking developments towards progress.

b) It is recommended that the DG Steering Committee consider the role expected of CESI in supporting policy development and federal government decision making, and explore additional means to increase CESI’s usefulness, such as possible linkages to the implementation of the 2008 Federal Sustainable Development Act (FSDA).

Response: The CESI DG Steering Committee agrees with the recommendation to consider the role expected of CESI in supporting policy development and federal government decision making. Beyond the specific reporting mandate of the initiative, the Strategic Information Integration Directorate will apply its experience with environmental indicator development to the creation of a departmental centre of expertise to provide guidance, policy advice and statistical analysis in support of the development of new indicators, as appropriate, to strengthen the information base for 1) department-wide activities such as the Departmental Performance Report or the Program Activity Architecture, as well as 2) government-wide activities such as the annual Canada’s Performance report and implementation of the 2008 Federal Sustainable Development Act (FSDA).

The revised CESI Indicators Logic Model, Performance Measurement Plan and data collection mechanisms, developed in response to Recommendation 1, will refine and clarify the intended target audiences. One audience is: federal government managers and staff with responsibility for the Federal Sustainable Development Strategy (FSDS). As a result, the CESI indicators will be the adopted indicators in the implementation of the 2008 Federal Sustainable Development Act by December 2009 in order to meet timelines laid out in the FSDA legislation.

1.0 INTRODUCTION

This report presents the results of the evaluation of the Canadian Environmental Sustainability Indicators (CESI) initiative conducted by Environment Canada’s Audit and Evaluation Branch between March and December 2008.

This evaluation focused on assessing the CESI initiative’s relevance, success, cost effectiveness and design and delivery since its inception in 2005. It relied on a review of documents, including recent external studies of the initiative’s relevance and effectiveness, and key informant interviews.

This document presents the findings and recommendations of the evaluation and is organized in the following way. Section 2 provides background information on the origins of the CESI initiative as well as a description of the structure of the current initiative. Section 3 describes the purpose of this evaluation and the methodology used, including the evaluation issues covered. Section 4 identifies the evaluation’s findings. Section 5 presents the conclusions based on the evaluation findings. Sections 6 and 7 contain, respectively, the recommendations and the management response to the recommendations.

2.0 BACKGROUND

This section presents an overview of the CESI initiative’s origins and provides a description of its main features.

2.1 Origins of the CESI Initiative[1]

The CESI initiative stems from two earlier initiatives. In 2000, the National Round Table on the Environment and the Economy (NRTEE) was working on ecological fiscal reform and requested funds to further pursue research on this topic. In response, the federal government allocated $9 million in the 2000 budget for two parallel initiatives: 1) the establishment of a Task Force on the Canadian Information System for the Environment; and 2) the Environmental and Sustainable Development Indicators (ESDI) Initiative.

Task Force on the Canadian Information System for the Environment (CISE)

The Task Force on the Canadian Information System for the Environment was created in October 2000 by the federal Environment Minister. The Task Force included members from a range of sectors, including consumers, university researchers, environmental activists, non-governmental organizations, municipalities, the private sector, government, and the international environmental community. Its purpose was to inform the design and implementation of an environmental information system that would provide a credible foundation for holding governments accountable; strengthen the basis for sound public policies; and enable Canadian citizens and organizations to adapt to environmental change and to play their individual and collective roles in environmental management.

At the end of its consultations, the Task Force recommended the creation of a Canadian Information System for the Environment (CISE) that would provide “environmental data to support a national set of sustainable development indicators, particularly those currently being developed by the National Round Table on the Environment and the Economy”.[2]

Environmental and Sustainable Development Indicators (ESDI) Initiative

The Environmental and Sustainable Development Indicators (ESDI) Initiative was launched by the NRTEE to develop a small set of high-level indicators that could link the environment and the economy and could be used to supplement economic indicators such as the gross domestic product (GDP). The initiative was steered by a committee formed of academics, experts on indicators and environmental accounting, business representatives, representatives of non-governmental organizations (NGOs) and staff from both Statistics Canada and Environment Canada.

In its final report in 2003, the NRTEE recommended that a small set of national natural and human capital indicators be reported annually, that Canada develop a revised System of National Accounts that would include information on natural and human capital, and that a better national network be created to monitor Canada’s environmental assets.

Following the Task Force and NRTEE recommendations, the CESI initiative was designed to produce indicators specifically designed for political decision makers and the general public and to be reported in the annual Federal Budget statement alongside the traditional economic indicators.

2.2 Description of the Initiative

CESI Goals

The Canadian Environmental Sustainability Indicators (CESI) initiative is a program initiated by the Government of Canada to develop and report on a small set of priority environmental indicators, notably air quality, water quality and greenhouse gas emissions. It is designed to supplement traditional social and economic measures, such as gross domestic product (GDP). The indicators are intended to help provide Canadians with a better understanding of the relationships that exist among the economy, the environment and human health and well-being with respect to air quality, water quality and greenhouse gas emissions. The indicators are also intended to assist those in government who are responsible for developing policy and measuring performance.[3]

The CESI initiative is based on the premise that Canadians need clearly defined environmental indicators -- measuring sticks that describe the health of their environment and track the results that have been achieved through the efforts of governments, industries and individuals to protect and improve the environment.[4] The CESI logic model and planning documents identified the target audience as members of the general public, Government of Canada managers and employees, decision makers and provinces, territories and municipalities.

The initiative’s intended ultimate outcomes are:

• Canadians recognize and use the Indicators as one source of trusted information on air, water, and greenhouse gases;
• Knowledge is shared effectively with target audiences and stakeholders are well served nationally;
• Canadians better understand how their actions affect the quality of the environment and human health; and
• Political and senior decision makers are more informed and make better decisions.

The complete CESI logic model is presented in Annex 1.

Statutory Context

The Minister of the Environment has statutory obligations under the Department of the Environment Act, the Canada Water Act, and the Canadian Environmental Protection Act, 1999 (CEPA 1999) to provide environmental information to Canadians.

In particular, the Department of the Environment Act (1971) states that:

“5. The Minister, in exercising his powers and carrying out his duties and functions under section 4, shall

• (a) initiate, recommend and undertake programs, and coordinate programs of the Government of Canada that are designed
  • (iii) to provide to Canadians environmental information in the public interest.”

Furthermore, CEPA 1999 states that:

“44. (1) The Minister shall

• (a) establish, operate and maintain a system for monitoring environmental quality;
• (b) conduct research and studies relating to pollution prevention, the nature, transportation, dispersion, effects, control and abatement of pollution and the effects of pollution on environmental quality, and provide advisory and technical services and information related to that research and those studies;
• (c) conduct research and studies relating to
  • (i) environmental contamination arising from disturbances of ecosystems by human activity,
  • (ii) changes in the normal geochemical cycling of toxic substances that are naturally present in the environment, and
  • (iii) detection and damage to ecosystems;
• (d) collect, process, correlate, interpret, create an inventory of and publish on a periodic basis data on environmental quality in Canada from monitoring systems, research, studies and any other sources;
• (e) formulate plans for pollution prevention and the control and abatement of pollution, including plans respecting the prevention of, preparedness for and response to an environmental emergency and for restoring any part of the environment damaged by or during an emergency, and establish, operate and publicize demonstration projects and make them available for demonstration; and
• (f) publish, arrange for the publication of or distribute through an information clearing-house
  • (i) information respecting pollution prevention,
  • (ii) pertinent information in respect of all aspects of environmental quality, and
  • (iii) a periodic report on the state of the Canadian environment.”

Funding

The funding for the initiative was set out in the 2004 federal budget and the initiative was formally approved in 2005. Treasury Board allotted $45 million for fiscal years 2005–2006 to 2008–2009 among Environment Canada ($32.5 million), Statistics Canada ($10.5 million) and Health Canada ($2 million), with an average of 53 full-time staff equivalents dedicated to the project each year. The current mandate for the initiative runs until the end of fiscal year 2008–2009.

CESI Products[5]

The CESI initiative reports on a yearly basis on three environmental indicators of key concern to Canadians: air quality, water quality and greenhouse gas emissions.

The air quality indicators track measures of exposure of Canadians to ground-level ozone and fine particulate matter (PM_2.5). These are key components of smog and two of the most pervasive and widely spread air pollutants. Exposure to these pollutants can be harmful. Both the ozone and PM_2.5 exposure indicators are population-weighted average concentrations observed at monitoring stations across Canada during the warm season (April to September).

The greenhouse gas emissions indicator tracks the annual Canadian releases of the six greenhouse gases (carbon dioxide, methane, nitrous oxide, sulphur hexafluoride, perfluorocarbons and hydrofluorocarbons) that are the major contributors to climate change. The indicator comes directly from the National Inventory Report prepared annually by Environment Canada for the United Nations Framework Convention on Climate Change.

The freshwater quality indicator uses the Water Quality Index endorsed by the Canadian Council of Ministers of the Environment to summarize the status of surface freshwater quality. Quality is assessed by examining the extent to which water quality guidelines for the protection of aquatic life (plants, invertebrates and fish) are being met at selected lake and river monitoring sites throughout Canada.

The CESI product suite currently[6] includes:

• Feature Report – CESI’s main product, the Feature report, presents the latest national status for each indicator, trends over time (for some indicators), an interpretation of the indicator results, a short description of influences that may have affected them, and plans for future improvements to the indicators. Where trend information is available, the main focus is on long-term trends, not annual fluctuations. The report concludes with a discussion of how the indicators are linked, primarily focusing on the socio-economic factors influencing the status and trends associated with the indicators.
• Highlights Report - The Highlights report shows the national situation and places more emphasis on the international context and on the important relationships that exist between the economy, the environment, and human health and well-being.
• Data Sources and Methods Reports - These three indicator-specific reports outline and explain the underlying methods and data for each indicator as published in the Feature report. The Data Sources and Methods reports provide technical detail and other background to facilitate interpretation of each indicator or allow others to conduct further analysis using the CESI data and methods as a starting point.
• Website - All CESI reports are available in web format for users to download and print on their personal computers from the CESI website. Furthermore, the website offers the following additional features:
  • An explanation of what each indicator covers;
  • Maps and charts for each indicator; and
  • Additional socio-economic information in the form of a supplementary set of tables compiled to accompany the environmental sustainability indicators. The main purpose of this publication is to provide contextual information on the human activities that have influenced the environmental indicator trends.

CESI Governance and Production Cycle

The CESI initiative is a collaborative effort of Environment Canada, Statistics Canada and Health Canada, with input from Canadian provinces and territories. Production of the CESI product suite is led by Environment Canada and entails a wide range of activities including scientific research; nationwide monitoring of environmental changes; assembling the data; refining, analyzing and calculating the indicators; writing, reviewing and revising the reports and website; and planning the next steps of the CESI initiative.

Under a tri-department Steering Committee composed of senior managers, the CESI Working Committee, chaired by Environment Canada and Statistics Canada, oversees the work of eight working groups (Reporting and Integration, Research and Development, Information and Architecture, Air Quality, Greenhouse Gas Emissions, Water Quality, Socio-Economic Modules, and Surveys). The Chairs of these working groups all sit on the CESI Working Committee. The following table outlines the CESI governance structure.

Table 1 CESI Governance Structure

The following table outlines the annual CESI production cycle:

Table 2 CESI Production Cycle

January– April	Work planning: Highlights report and other products (content, design and structure) – outlines, mock-ups
February– June	Website redesign (design outline, grey model)
January– May	Indicator development improvements for upcoming year
February– June	Data acquisition, calculation and quality assurance / quality control for each indicator
March– June	Surveys and socio-economic data integration into indicator sections
April– November	Writing and first draft of product suite
June– November	Site level data (metadata and drill down capability)
November– December	Scientific & technical and provincial review and revisions
December– January	Management reviews, approvals and final revision
December– February	Editing, translation and layout of product suite
November– February	Website (design, tools, testing, content)
November– February	Communication, engagement, marketing, briefing

3.0 METHODOLOGY

3.1 Approach

This evaluation was conducted in two separate phases. First, a review of secondary data was done. Based on the results of the secondary data analysis, key informant interviews with CESI managers[7] were then conducted.

To conduct the secondary data analysis, a list of evaluation issues was developed, based on standard evaluation questions. For each evaluation issue, relevant documents were reviewed and key findings summarized in separate findings tables. A conclusion was then drawn on whether available secondary evidence adequately addressed each evaluation issue. A description of key secondary sources can be found in Annex 2 and the full list of secondary sources reviewed is presented in Annex 3.

The large number and diversity of secondary sources available on the CESI initiative’s relevance and performance enabled the evaluation team to adequately address most evaluation issues examined as part of this evaluation. However, two gaps were found that required additional, primary data to be collected as part of a second phase for this evaluation. The first gap pertained to the success of the initiative in achieving or being on track to achieving its expected immediate and intermediate outcomes (evaluation issue number 3). The second gap pertained to the initiative’s governance and accountability framework (evaluation issue number 7).

In order to fill these gaps, key informant interviews of selected CESI managers were conducted. These interviews were conducted by Phoenix Strategic Perspectives Inc. on behalf of Environment Canada during the period of October 2–29, 2008. Eleven interviews in total were conducted with CESI managers from Environment Canada, Health Canada and Statistics Canada. Interview respondents all belonged to one of the three following groups:

• CESI Steering Committee (three respondents)
• CESI Working Committee (four respondents)
• CESI working groups (four respondents)

The two main issues that were explored as part of these semi-structured interviews were questions on the achievement of CESI outcomes, specifically the recent changes made to CESI products and solutions implemented to improve its efficiency, and the adequacy and effectiveness of the CESI governance and accountability structure. As part of these interviews, the consultant asked interview participants for any additional documentation relevant to the evaluation. One document was received as part of this process. Annex 4 includes the interview guide that was used to conduct the key informant interviews.

3.2 Evaluation Issues

The following evaluation issues were addressed as part of this evaluation:

Relevance

1. Is there a legitimate and necessary role for government in this program area or activity?

2. Is the initiative connected with societal/environmental needs?

Success

3. To what extent has the CESI initiative achieved (or is on track to achieving) its expected immediate and intermediate outcomes for components for which Environment Canada is responsible or shares responsibility?

4. Have there been any unintended (positive or negative) outcomes? Were any actions taken as a result of these?

Cost Effectiveness

5. Are others involved in the same areas of activities and/or share similar objectives? How is duplication avoided and complementarity achieved?

6. Is the initiative delivered in the most efficient manner?

Design and Delivery

7. Is the CESI initiative governance effective?

8. Is performance data collected against activities/outcomes? If so, is collected information used to inform senior management / decision makers?

9. Does the initiative identify clear deliverables and expected results? Is the initiative delivered as designed?

3.3 Limitations of the Evaluation

This evaluation relies on multiple sources but is limited to two lines of evidence, a review of documents and key informant interviews with CESI managers, thereby limiting evaluators’ capacity to triangulate evaluation findings.

This limitation was, however, mitigated by the rich available documentation on the initiative’s performance. The following three documents were particularly useful:

• International Institute for Sustainable Development (IISD). Review of the Canadian Environmental Sustainability Indicators Initiative. Final Report. April 3, 2008: This external study targeting senior-level policy makers and experts in accountability reporting addresses their perception of CESI’s relevance and usefulness. A total of 13 interviews were conducted.
• Phoenix Strategic Perspectives Inc. Final Report. Canadian Environmental Sustainability Indicators Initiative. User Needs Assessment and Product Testing. December 2007: This external study targeting industry, NGOs and members of the Canadian public addresses their perception of CESI’s relevance and usefulness. The research included four focus groups made up of members of the general public with an interest in environmental issues, and 24 one-on-one interviews with stakeholders from other target audiences, including environmental non-governmental organizations (ENGOs) and industry. Members of the Canadian public invited to participate in this study were recruited randomly among members of the adult populations of Montreal and Toronto. Participants were screened to only include those who reported having an interest in environmental issues and having looked for environmental information during the previous year.
• Environment Canada. Strategic Information Integration Directorate (SIID). CESI Review: Issues and Challenges: In this internal review of CESI implementation, CESI managers identified progress towards intended CESI deliverables, flagged challenges, and proposed potential solutions.

4.0 FINDINGS

Below are the findings of this evaluation presented by evaluation issue (relevance, success, cost effectiveness, design and delivery) and by the related evaluation questions as presented in Annex 2. The findings at the overall issue level are presented first, followed by the findings for each evaluation question.

A rating is also provided for each evaluation question. The ratings are based on a judgment of whether the findings indicate that

• The intended outcomes or goals have been achieved or met – labelled as Achieved;
• Considerable progress has been made to meet the intended outcomes or goals, but attention is still needed - labelled as Progress Made, Attention Needed; or
• Little progress has been made to meet the intended outcome and attention is needed on a priority basis – labelled as Little Progress, Priority for Attention.
• The N/A symbol identifies items where a rating is not applicable.

A summary of ratings for the evaluation issues and questions is presented in Annex 5.

Evaluation Issue 1: Relevance

Overall Findings: Available data show that the CESI initiative was and remains relevant to both federal government priorities and environmental needs. Documents indicated that there is a clear role for federal government in producing environmental sustainability indicators, that an appropriate partnership was developed across federal departments and provincial/territorial governments to deliver this initiative, and that it addresses a recognized need for macro-economic environmental sustainability indicators among the Canadian public and government policy and decision makers.

Evaluation Issue: Relevance	Indicator(s)	Methods	Rating
1. Is there a legitimate and necessary role for government in this program area or activity?	• Demonstration of a clear/legislated mandate for federal government intervention • Congruence with federal government priorities • Stakeholder support for federal government intervention	• Document Review	Achieved

Numerous sources underline the importance of the federal government’s involvement in the production of broad environmental indicators.

• According to documents reviewed, there is a clear/legislated mandate for federal government intervention in the production of environmental indicators. The main authority for such intervention comes from the Canadian Environmental Protection Act, 1999 under which the federal Minister of Environment has legal responsibility to produce and disseminate information on the state of the environment. CEPA requires the Minister to “establish, operate and maintain a system for monitoring environmental quality.”[8] In keeping with this legislated mandate, the CISE Task Force recommended in 2001 that the federal government jointly sponsor with the provincial and territorial governments an institute responsible for measuring sustainability indicators.[9]
• Budget announcements in 2000 and 2004 provide evidence of congruence of the initiative with federal government priorities at the time the CESI initiative was created. Funding was announced in Budget 2000[10] to support collaboration between Environment Canada, the National Round Table for the Environment and the Economy (NRTEE) and Statistics Canada to develop indicators to measure environmental performance in conjunction with economic performance, while Budget 2004[11] announced funding and support for environmental technologies and remediation of contaminated sites.
• A user needs assessment conducted by Phoenix Strategic Perspectives Inc. in 2007 also revealed support among members of the Canadian public for a strong federal role in this field. They strongly endorsed the importance of the CESI initiative to provide Canadians with information that supports credible, informed policy decisions and good governance.[12]

Evaluation Issue: Relevance	Indicator(s)	Methods	Rating
2. Is the initiative connected with societal/ environmental needs?	• Demonstration that the CESI initiative addresses identified environmental/societal needs	• Document Review	Achieved

Documentation shows that stakeholders (NRTEE, Task Force on the Canadian Information System for the Environment and general users) agree with the range of indicators covered by the CESI initiative, the need for environmental sustainability indicators to inform decision making, policy development and adaptation to environmental changes, and for a strategic approach to producing such indicators. In relying on a strategic partnership between Environment Canada, Health Canada and Statistics Canada to produce broad indicators linking environmental sustainability to the economy, the CESI initiative addresses the expressed needs.

• In 2001, the Task Force on the Canadian Information System for the Environment reported that Canada lacked a strategic and integrated approach to collecting environmental information, and that, as a result, Canadians lacked key environmental information. To fill this need, the Task Force recommended that the Government of Canada provide environmental information through the creation of a Canadian Information System for the Environment (CISE) in support of a national set of sustainable development indicators[13].
• In 2003, NRTEE identified the need for national environment and sustainable development indicators to supplement the macroeconomic indicators that currently help summarize the state of the national economy to “provide Canadians with a more robust picture of the state of our national capital”. Their list of recommended indicators included, among others, air quality, freshwater quality, and greenhouse gas emissions.[14]
• The CISE website identified environmental indicators as one of the tools essential to modern environmental management that should form the basis for effective public policies (to help identify and prioritize policy issues and help to assess the effectiveness and efficiency of environmental policies and programs)[15].
• In 2007, CESI staff distributed a questionnaire to 715 potential respondents aiming to measure the perceived usefulness of the 2006 and 2007 CESI publications. A majority (74%) of the 71 individuals who responded to the survey said they intended to make use of the CESI publications.[16]
• Interviews with members of the general public and non-government stakeholders (i.e., industry and NGOs) as part of the assessment of user needs by Phoenix Strategic Perspectives Inc. revealed unanimous concern over environmental sustainability, and most respondents found the indicators appropriate and relevant, especially data on greenhouse gases. Respondents also believed the CESI indicators to be necessary in order to adopt best practices, establish benchmarks, foster healthy competition, enhance government credibility, enhance government accountability, and increase interconnectedness with other countries.[17]
• Interviewees (i.e., senior level policy makers and experts in accountability reporting) from the 2008 International Institute for Sustainable Development (IISD) review of the CESI initiative agreed with the indicators covered by the initiative but felt that data needed to be made available at a smaller, more local scale to better feed policy making and to be useful for accountability purposes. They also highlighted the need for better linkages to issues of interest to the general public and Members of Parliament (such as human health issues).[18]
• The 2006 Standing Committee on the Environment and Sustainable Development responsible for the review of CEPA’s provisions and operation confirmed the need for “timely, accurate and accessible environmental information, integrated with socio-economic factors, to improve decision making and support progress towards sustainability” and recommended that “the government publish biennially, in electronic and hard copy formats, a comprehensive state of the environment report.”[19]

Evaluation Issue 2: Success

Overall Findings: While the CESI initiative has produced its intended outputs and achieved or is on track to achieve most of its outcomes pertaining to increased capacity to produce environmental indicators, no evidence was found that CESI products are actually being used, except by Treasury Board in its annual Canada’s Performance report and by Human Resources and Social Development Canada in its set of human well-being indicators. Attention is needed with respect to achieving intermediate outcomes targeting the use of CESI information by decision makers and the general public, as well as the integration of CESI results in policy decisions. To address this limitation, CESI managers are currently making enhancements to the CESI website and products such as increased drill-down capacity and the integration of socio-economic information. The program’s information dissemination strategy also appears to require improvements, in light of target users’ generally low awareness of CESI products. Furthermore, the use of CESI for government decision/policy making is seen as being somewhat dependent on a federal government commitment to specific and measurable objectives related to the CESI indicators.

Evaluation Issue: Success	Indicator(s)	Methods	Rating
3. To what extent has the CESI initiative achieved (or is on track to achieving) its expected immediate and intermediate outcomes for components for which Environment Canada is responsible or shares responsibility?	Immediate Outcomes Evidence of: 1. more targeted partnerships and distribution of funds 2. provinces being equipped to publish their own environmental reports 3. a coordinated national approach 4. improved collaboration mechanisms 5. engagement (data exchange) by all provinces and federal departments 6. increased recognition for on-going environmental monitoring programs Intermediate Outcomes re Practices Evidence of: 1. CESI results being reported to Canadians 2. government actions in response to CESI results 3. indicator tools/approaches being used in policy/decision making (by governments and general public) Intermediate Outcomes re Capacity 1. Evidence of increased capacity of the CESI initiative to meet its intended outcomes (i.e., information system in place; improved and expanded monitoring systems; drill-down capacity at site, local, regional and provincial levels; improved timeliness of information dissemination)	• Document Review • Key Informant Interviews	Progress made; Attention needed

All sources suggest that the CESI initiative has produced its intended outputs and achieved or is on track to achieving most of its outcomes pertaining to increased capacity to produce environmental indicators The initiative, however, appears to be less successful in reaching and addressing the needs of its target clienteles (general public, non-government organizations and government policy developers and decision makers). To address this limitation, CESI managers are currently making enhancements to the CESI website and products such as increased drill-down capacity and the integration of socio-economic information. The program’s information dissemination strategy also appears to require improvements, in light of target users’ generally low awareness of CESI products. Furthermore, the use of CESI for government decision/policy making is seen as being somewhat dependent on a federal government commitment to specific and measurable objectives related to the CESI indicators.

Immediate Outcomes

1) More targeted partnerships and distribution of funds:

• Key informant interviews with CESI representatives revealed that targeting of partnerships and distribution of funds are addressed satisfactorily on an annual basis. Examples provided to support positive achievements in this area include the following:
  • Funding to improve monitoring of air quality, including the improvement of existing monitors and the purchase of PM_2.5 monitors
  • Funding to deal with costs incurred by the provinces/territories to collect data on water quality
  • Funding to develop bilateral agreements with some provinces to improve the provision of water quality data in a timely manner
  • Work on development of an air health risk indicator with Health Canada

2) Provinces being equipped to publish their own environmental reports:

• Nearly half of the CESI key informant interview respondents suggested that provinces already have the capacity to publish their own environmental reports but that the extent to which this is being done varies across the provinces (Newfoundland and Québec were cited as good examples of provincial reporting). Other respondents suggested that this is not really a goal of the CESI initiative, but that through capacity building (i.e., the initiative has helped to build monitoring capacity) and provincial-federal partnerships, provinces’ ability to publish such reports has improved.

3) Coordinated national approach and 4) Improved collaboration mechanisms in place and 5) Engagement by all provinces and federal departments:

• CESI key informants indicated that, at the federal level, there exists a coordinated approach and good collaborative mechanisms. The CESI initiative was cited by several interviewees as a good example of interdepartmental coordination and collaboration, this success being attributed in part to the governance structure of the CESI initiative.
• All CESI interviewees felt that federal and provincial coordination and collaboration mechanisms were good and/or improving. Specific examples provided to support this impression include the following:
  • Agreements between the federal government and the provinces have been signed for monitoring and processing of water quality information. Every province has agreed to make their water data available this year.
  • Work is being done with the provinces to ensure consistency in interpreting and reporting water quality data and there is a proposal to simplify the water quality index (WQI) to a core set of parameters that are common to most or all watersheds/monitoring stations.
  • Each year since the CESI initiative began there has been a national CESI federal-provincial water quality index conference/workshop in coordination with the Canadian Council of Ministers of the Environment (CCME) Water Quality Task Group.
  • The CESI air group coordinates with the National Air Pollution Surveillance (NAPS) and Canadian Air and Precipitation Monitoring Network (CAPMoN) groups through the CCME federal-provincial air monitoring coordinating committee.
• There was a consensus among CESI interview respondents that there is still work to be done in the area of water monitoring in order to ensure nationally representative water quality data.

6) Increased recognition for ongoing environmental monitoring programs:

• Members of the general public consulted as part of the Phoenix user needs assessment showed support for environmental monitoring initiatives by stating that they actively seek information on the state of the environment and that they more readily trust environmental reports that are based on data. They stated that an initiative such as CESI is important because awareness of sustainability issues is the basis for addressing them and because such data are required for accountability and transparency purposes.[20]
• Representatives of non-government organizations consulted as part of the Phoenix user needs assessment also reported that they routinely use environmental sustainability data. They emphasised the importance of ongoing tracking of state-of-the-environment data.[21]

Intermediate Outcomes Regarding Practices

1) CESI results being reported to Canadians:

• Results from the survey of recipients of the 2007 Highlights report showed that only 59% (42 of 71) of respondents were aware of the CESI initiative prior to receiving the 2007 Highlights report and only 35% (25 of 71) of respondents recalled receiving copies of the CESI 2006 Highlights report.[22]
• Few non-government stakeholders consulted as part of the Phoenix needs assessment study were aware of the CESI products prior to being contacted for this research. Several members of the general public who took part in the same research stated that CESI products need to be promoted or advertised so that Canadians are aware of them.[23]
• In spite of the above findings, there was consensus among CESI key informants that the initiative is achieving and performing well in reporting CESI results to Canadians by meeting its reporting timelines, making reports and information available, and more recently through the redesign of the CESI website to make information more accessible to Canadians and to allow them to choose the level of detail they want.
• Some CESI interview respondents, however, felt that more could be done to increase awareness of CESI among members of the general public. Specifically, they felt that release of CESI reports could be publicized more by putting out more media advisories.
• Program documents indicate that communication strategies were developed for the 2005, 2006, 2007 and 2008 releases of the CESI products. These strategies generally recommended a “low-profile” communications approach to target stakeholders. Target stakeholders included environmental groups, parliamentarians, federal and provincial/territorial policy managers, specialists and analysts. The principal means of dissemination consisted of briefing packages distributed to policy/decision makers and other target stakeholders, posting of the CESI products on various websites, and posting of an official bulletin on Statistics Canada’s publication The Daily.
• A number of improvements are currently being made to the CESI website and products (to be made available with the publication of the 2008 CESI products) to improve their accessibility for Canadians and all potential users. These changes include the following:
  • Improved drill-down capacity (i.e., the ability to access data at site/local/regional levels).
  • Shift from lengthy, descriptive reports to a dynamic online information source (the Feature report will be discontinued in favour of a searchable online database).
  • Inclusion of trend information over time (this feature is still embryonic but will expand as sets of data are gathered for each indicator on an annual basis).
  • Better integration of socio-economic context information in the CESI products instead of presenting it as a separate section.
  • Redesign of the 2008 Highlights report to more explicitly target the general public by ensuring it is written in clear, understandable, and meaningful language.[24]

2) Government actions in response to CESI results and 3) Indicators tools/approaches are being used in policy/decision making (by governments and general public):

• Senior level policy makers and experts in accountability reporting interviewed as part of the ISSD review indicated that CESI indicators are not being used for policy development or decision making namely because they require detailed information at a scale and a level of precision that a national, contextual set of indicators such as those provided by the CESI initiative do not provide. To address some of the perceived weaknesses which impede the indicators’ use, the IISD report recommended:
  • reporting CESI indicators at a more disaggregated level;
  • providing more contextualization of the CESI indicators within the larger and environmental and health context of Canada;
  • providing trend information (i.e., data over a number of years); and
  • using CESI data for accountability reporting (i.e., using CESI indicators as measures of the federal government’s achievement of specific intended outcomes and specifying which jurisdictions are accountable for which achievements).[25]
• CESI managers recently interviewed on this topic generally agreed with the above observations and reported that they are in the process of addressing these issues. As previously mentioned, the 2008 CESI data will be available at the site level (drill-down capacity) using an improved website navigation capability. Data will also be better contextualized within the larger and environmental and health context of Canada by integrating socio-economic and related information into the reports themselves rather than as part of a separate document.
• CESI key informants, however, commented that trend information will not be available until a few years of data have been gathered systematically and that CESI’s ability to inform concrete and specific policy-related action and decision making will depend on the effective linkage of CESI indicators with socio-economic indicators in future reports.
• Some CESI interview respondents also explained that while the original idea behind CESI was to develop environmental indicators similar to economic ones like Gross Domestic Product, the initiative is actually designed to provide contextual and background information to inform people and increase environmental literacy. Its current purpose is not seen to be informing program or policy development.
• Finally, a few respondents commented that CESI could be more effective in relation to these outcomes if the federal government committed itself to specific and measurable objectives related to the CESI indicators. In short, the point being made was that government needs to take the lead in this regard through an express willingness to use CESI information for decision making.
• CESI indicators are being used by Treasury Board in its annual Canada’s Performance report and by Human Resources and Social Development Canada in its set of human well-being indicators.

Intermediate Outcomes Regarding Capacity

1) Increased capacity of the CESI initiative to meet its intended outcomes:

Information systems in place to support the three indicators

• The ISSD Review reported that, according to senior level policy makers and experts in accountability reporting, effective information systems were in place to support the three CESI indicators.[26]
• CESI representatives also agreed that there are effective and efficient systems in place, particularly the data collection systems, to support the three indicators. The systems in place for greenhouse gas emissions and air quality were described as good or very good, while the system for water monitoring was described as lagging a little behind the first two but improving.
• A few CESI interview respondents referred to the CESI website and enhanced web reporting as a good information system in place to support the three indicators, and specifically to the ability to drill down to local or regional information. That said, it was also suggested that there should be more interface between the CESI website and Statistics Canada’s website, especially because of the desire to integrate socio-economic information with the CESI indicators.
• Reference was also made by CESI key informants to Statistics Canada’s water usage surveys as part of the information system in place to support the water quality indicator. This included the agriculture water use survey, the industry water use survey, and the drinking water survey.

Monitoring systems improved and expanded in areas of air, water, greenhouse gases

• All CESI interview participants agreed that this outcome is being achieved, while also recognizing that there is room for improvement, especially in the area of water monitoring in order to ensure nationally representative water quality data.

Drill-down capacity (per site/local/regional/provincial)

• Most members of the general public and non-government stakeholders (i.e., industry and NGOs) consulted as part of the Phoenix user needs assessment found that CESI information was not sufficiently detailed to meet their needs.[27]
• This observation was echoed by senior-level policy makers and experts in accountability reporting consulted as part of the ISSD review. They specifically mentioned drill-down capacity as a key requirement for CESI data to be useful for program and policy development.[28]
• Recent CESI senior management briefings have highlighted current plans for the CESI initiative to transition to a new State of the Environment Reporting vision which addresses most of the concerns listed above. As part of this new vision, the CESI 2008 report will provide more trends, access to site level information, better integration of socio-economic information, and enhanced web-based reporting. The web design will enable access to local and regional level information, customized information and search capacity.[29]
• CESI key informants confirmed that changes are currently being brought to the 2008 CESI products and website to address the above-mentioned concerns. While interview respondents clearly saw this as a positive development in general, nearly half indicated that the capacity to drill down, while resolving one problem (i.e., lack of access to local/regional data), gives rise to others. They identified specific problems: the fact that the provinces were reluctant to provide this kind of information as some prefer that it be accessed through their own websites and that providing these data in their raw form could potentially lead to mistaken interpretations of their meaning since there may be additional factors that have to be taken into account in order to properly interpret the data.

Ability to deliver information in a more timely fashion

• An internal review conducted by CESI management at the end of 2006 revealed that although timelines and expectations of the CESI production process were generally met, a number of challenges were encountered, such as the late receipt of the indicators data and the resulting pressures to develop CESI products.[30]
• CESI key informants reported that the situation has improved over time but recognized that there is a structural limit to improving timeliness that will never be overcome. Specifically, it was acknowledged by all that there would always be a lag between the time the data are collected and interpreted and the time the data are published or reported. The current two-year lag was seen by all as something that could be improved, but a one-year lag was the limit of what was seen to be achievable.
• Factors identified as having already contributed to or likely to contribute to more timely delivery of information included ceasing production of the CESI Feature report, improved reporting time on the air quality indicator, and increased use of the website as a delivery mechanism.
• Respondents to ISSD interviews indicated that the timeliness of CESI reports is adequate and the interviews did not reveal frustration with the lack of timeliness of the data on the website.

Evaluation Issue: Success	Indicator(s)	Methods	Rating
4. Have there been any unintended (positive or negative) outcomes? Were any actions taken as a result of these?	• Evidence of unintended outcomes	• Document Review	N/A

No significant unintended negative or positive outcomes appear to have resulted from the initiative.

Evaluation Issue 3: Cost Effectiveness

Overall Findings: In light of the findings outlined above on the additional activities currently under way to ensure the success of the initiative, it was deemed premature to conduct a full cost-effectiveness analysis. Rather, for the purpose of the current evaluation, cost effectiveness was measured by examining the issues of duplication, complementarity and efficiency. The CESI initiative appears to be delivered in a generally efficient manner. Duplication is avoided through collaboration with other federal departments and provincial governments, and complementarity is achieved through the use of the CESI indicators by Treasury Board in its annual Canada’s Performance report and Human Resources and Social Development Canada in its set of human well-being indicators.

Evaluation Issue:	Indicator(s)	Methods	Rating
5. Are others involved in the same areas of activities and/or share similar objectives? How is duplication avoided and complementarity achieved?	• Evidence of duplication/complementarity	• Document Review • Key Informant Interviews	Achieved

While other sources of data exist, the CESI initiative appears to have a unique mandate and to involve collaboration among the key government stakeholders.

• By its nature, CESI is a collaborative initiative between the three participating departments (Environment Canada, Health Canada and Statistics Canada).
• Although some duplication between CESI and other sources of data was identified by non-governmental stakeholders, the same stakeholders pointed to the need for official data on some key indicators of relevance to the Canadian public and decision makers and to the federal government as being the most appropriate provider of such data.[31]
• CESI key informants also identified what they consider to be other sources of indicators and data related to Canada’s environmental sustainability. These include: Environmental Signals; the National Pollutants Release Inventory (NPRI); the Clean Air website; the Greenhouse Gas Inventory; and the Canadian Council of Forest Ministers’ Criteria and Indicators of Sustainable Forest Management in Canada. While not related specifically to Canada, the Organisation for Economic Co-operation and Development (OECD) and the Intergovernmental Panel on Climate Change (IPCC) were also mentioned. In the absence of a commonly accepted framework for measuring sustainability, they observed that it is currently difficult to avoid a multiplication of sources of indicators. However, they commented that complementarity is achieved through the use of the CESI indicators by Treasury Board in its annual Canada’s Performance report and Human Resources and Social Development Canada in its set of human well-being indicators.
• Senior level policy makers and experts in accountability reporting who were consulted as part of the ISSD review reported that the CESI indicators generally complement or are used in conjunction with other sources of data. They saw the use of CESI indicators for Canada’s performance reporting as recognition of their unique role.[32]

Evaluation Issue: Cost Effectiveness	Indicator(s)	Methods	Rating
6. Is the initiative delivered in the most efficient manner?	• Evidence of need for increasing efficiency	• Document Review • Key informant interviews	Achieved

While some areas for improvement were identified, no evidence was found of a serious need for increasing the initiative’s efficiency.

• The internal review conducted by CESI management at the end of 2006 identified the following implementation challenges:
  • the late receipt of the indicators data and the resulting pressures to develop CESI products on time;
  • pressures for earlier release of information;
  • ongoing improvements required of the indicators and the need to devote resources and time to longer-term research and indicator development for air and water; and
  • slowness in establishing agreements with provinces and territories to deliver the Water Quality Index.[33]
• As a result, the following required improvements were identified: 1) streamlining processes and improving coordination of activities in order to more easily meet a very tight reporting cycle; 2) re-examining distribution of roles and responsibilities and resourcing of activities across participating departments; and 3) refining agreements and communications with provinces.
• CESI representatives interviewed in fall 2008 were generally satisfied with the efficiency of the initiative’s implementation and found that overall processes had improved. The few remaining areas for improvement identified were as follows:
  • The governance structure is very time consuming: while ensuring effective functioning, the consensus decision-making approach, the number of committees and the frequency of meetings require a lot of time and effort.
  • Production deadlines remain tight, and strict respect for deadlines thus needs to be maintained.
  • All monies targeted for the CESI initiative must be used for CESI.
  • The need to report annually reduces the capacity to innovate in the area of indicators. Publishing every two to three years might be a better option.

Evaluation Issue 4: Design and Delivery

Overall Findings: The CESI initiative’s governance and accountability structure is clear and effective. Some performance data collection and reporting activities also appear to be adequate for documenting program performance and supporting decision making. These data were used by management to make design and delivery improvements, as evidenced by the changes being made to the 2008 product suite. However, the initiative lacks systematic means of effectively monitoring whether its target users are accessing and using the CESI information. While clear deliverables and expected results are documented and the CESI initiative is being delivered as designed, some CESI representatives disagree on the degree to which it is realistic to expect CESI to inform policy development and government decision making given that a federal government commitment to specific and measurable objectives related to the CESI indicators is currently lacking.

Evaluation Issue: Design and Delivery	Indicator(s)	Methods	Rating
7. Is governance of the CESI initiative effective?	• Evidence of clear definition of roles and responsibilities • Evidence of effective coordination, management and reporting mechanisms	• Document Review • Key Informant Interviews	Achieved

Various sources of evidence suggest that the governance and accountability structure for CESI is clear and effective; however, a few minor factors detract from the effectiveness of the initiative’s governance, such as time-consuming procedures for consensus-making and report approvals.

• A review of files and documents showed that the CESI initiative has a clearly developed and up-to-date governance structure, which depicts the various relationships between the working groups, secretariat and steering committees and identifies the individuals leading these various groups.[34]
• As well, working documents demonstrate periodic reviews of progress and regular monthly meetings among the various CESI teams and committees (Reporting and Analysis team; Indicators leads; Working Group Committee; inter-departmental) that highlight roles, responsibilities and target dates for various milestones for the development of the CESI reports. These documents provide descriptions of issues and challenges facing the CESI teams, with the assignation of roles for the delivery of agreed-on action items.[35]
• CESI key informant interviews also confirmed that the CESI initiative’s governance is effective. This effective governance is attributed to two factors: the maturity of the initiative and the good functioning of the Working Committee and Working Groups.
• Several CESI interview participants suggested that the governance structure’s effectiveness can be in part attributed to the way it engages individuals and to the requirement for consensus. Consensus, conversely, is also listed as a factor that makes the process time-consuming.
• Other weaknesses in the governance of CESI included an overemphasis on hierarchy and a time-consuming sign-off procedure where the report has to be signed off all at once instead of in stages.

Evaluation Issue: Design and Delivery	Indicator(s)	Methods	Rating
8. Are performance data collected against activities/ outcomes? If so, is collected information used to inform senior management / decision makers?	• Adequate performance data are available for all CESI activities • Evidence of decision making based on available performance data	• Document Review • Key Informant Interviews	Progress made; Attention needed

Some performance data collection activities appear to be adequate for documenting performance and supporting decision making, but the initiative lacks systematic means of effectively monitoring whether its target users are accessing and using the CESI information.

• Over the past few years, the initiative has produced numerous documents highlighting the initiative’s performance and identifying needed improvements. These include a log of the various products/events where CESI results were specifically cited or used in meeting a departmental obligation, reviews on progress toward the achievement of CESI deliverables, meeting summaries and records of decision, a survey of target users on the usefulness of publications, satisfaction surveys with the three CESI reports, a user-needs assessment study and interviews with potential government CESI users.
• The records of decision and the improvements being made to the 2008 CESI products and website are evidence that some of this performance information was used by senior management for decision-making purposes.
• Despite the above-mentioned performance measurement activities, the CESI initiative has no systematic means of effectively monitoring whether its target users are accessing and using the CESI information. In particular, there is no evidence that its database of potential users is representative of the target population for CESI products and an insufficient response was received to target user survey attempts.
• As per commitments, Statistics Canada conducted satisfaction surveys in 2006 and 2007 to determine if the CESI reports have been successful. Survey candidates were those who had indicated an interest in participating in an evaluation of the electronic publications during a pre-solicitation for future research. In all, 64 candidates were contacted in May 2006 (11 responded) and 81 in 2007 (18 responded). The survey was posted on a website and was communicated by email. It asked about satisfaction with the three CESI reports; reasons for use; most valued features; and suggestions to improve content and presentation. It used 5-point response scales. However, the survey population, number of respondents and cell sizes were too small for the findings to be useful.
• CESI staff distributed a separate questionnaire with the 2007 Highlights report. It aimed to measure the perceived usefulness of the 2006 and 2007 CESI publications. It was sent to 715 potential respondents. Only 71 recipients (10%) responded to the survey.
• CESI interview respondents identified some performance data strategies that could be employed in the future, including a “Your feedback” section on the CESI website, measuring the number of “hits” on the CESI website (and the pages visited), public opinion research on user needs and satisfaction with CESI products, and annual consultations/workshops with provincial partners.

Evaluation Issue: Design and Delivery	Indicator(s)	Methods	Rating
9. Does the initiative identify clear deliverables and expected results? Is the initiative delivered as designed?	• Documented deliverables and expected results • Congruence between intended and reported activities, deliverables and outputs	• Document Review • Key informant interviews	Progress made; Attention needed

The initiative identifies clear deliverables and expected results and is being delivered as designed. However, there is no consensus among CESI representatives on whether CESI indicators should be expected to inform policy development and decision making.

• The CESI main planning document clearly outlines the expected CESI deliverables and intended outputs and results.
• While CESI progress reports show agreement on the expected deliverables, various sources, however, showed a lack of agreement on the appropriateness of some of the outcomes outlined in the CESI logic model. Particularly, some CESI representatives disagreed with the expectation that CESI indicators be used for policy and decision making, given the absence of a federal government strategy and clear targets and accountabilities for environmental sustainability. As previously mentioned, some CESI interview respondents believed that the CESI initiative was not designed to inform program or policy development in the first place, but rather that it is actually designed to provide contextual and background information to inform people and increase environmental literacy. Others argued that the CESI initiative’s relevance, usefulness and actual use by federal policy makers would increase if the indicators could be used for accountability reporting purposes. This perspective was echoed by the senior-level policy makers and experts in accountability reporting consulted as part of the ISSD review.[36]
• A few CESI key informants also disagreed with the notion that equipping provinces to publish their own environmental reports should be within the purview of the CESI initiative.
• CESI internal reviews and testimonies from CESI representatives show that the initiative is being delivered as designed. Recent changes to the CESI deliverables remain in keeping with the original objective of the initiative, which is to report annually on three environmental indicators of key concern to Canadians: air quality, water quality and greenhouse gas emissions.

5.0 CONCLUSIONS

Overall, evidence shows that the CESI initiative is relevant to both federal government priorities and various stakeholders’ environmental information needs. While other similar sources of environmental sustainability data exist, the initiative appears to have a unique mandate and to involve collaboration among the key government stakeholders (i.e., Environment Canada, Health Canada, Statistics Canada, and the provinces and territories).

However, while the CESI initiative has produced its intended outputs and achieved or is on track to achieving most of its outcomes pertaining to increased capacity to produce environmental indicators, no evidence was found that CESI products are actually being used, except by Treasury Board in its annual Canada’s Performance report and by Human Resources and Social Development Canada in its set of human well-being indicators. Attention is needed with respect to achieving intermediate outcomes targeting the use of CESI information by decision makers and the general public, as well as the integration of CESI results in policy decisions.

In 2007, CESI managers contracted two external studies aiming, among other objectives, to measure the awareness and perceived usefulness of CESI products by CESI target users, namely, senior level policy makers, experts in accountability reporting, members of the general public with an interest in environmental issues, and representatives from ENGOs and industry. Results from these studies revealed that CESI products were not well known and had design flaws that limited their usefulness.

In order to address these shortcomings, and based on recommendations from these two studies, CESI managers are currently making enhancements to the CESI website and 2008 suite of products. These improvements include presenting CESI information in more common language, giving access to data at more specific levels (drill-down capacity), providing more trend information, and better integrating socio-economic and health context information.

One recommended change could, however, not be implemented. Several federal government policy makers and a few CESI representatives argued that the use of CESI data by federal government decision makers and policy makers would increase if it could be used for policy development and accountability reporting purposes. This is, however, seen as being dependent on the development of a federal environmental sustainability strategy and the identification of specific objectives against which the federal government would be measured using CESI indicators. No such objectives currently exist.

In light of the improvements that are currently being made to the initiative, it is too early to draw definitive conclusions on the adequacy of the CESI initiative to meet its target users’ need. Once these changes are fully implemented, the initiative will need to demonstrate its usefulness by documenting who its target users are and by monitoring the extent to which CESI products are known, accessed, used and considered useful by these target users.

The initiative currently lacks a performance measurement system to effectively monitor whether its target users are accessing and using the CESI information. The number of responses to the two user surveys conducted by Statistics Canada was too low for the results to be meaningful. Furthermore, evidence shows that there is generally low awareness of CESI products among the general public and non-government stakeholders, suggesting that a more effective information dissemination strategy may also be required.

Nevertheless, the initiative is generally implemented as designed and in an efficient manner. It benefits from an appropriate and effective governance structure that reflects the interdepartmental nature of its mandate and activities. CESI managers have also demonstrated a best results-management practice in producing meeting minutes, records of decisions and a review of CESI implementation challenges that led to effective solutions being developed.

6.0 RECOMMENDATIONS

The following recommendations are based on the findings and conclusions of the evaluation.

Recommendation 1: In order to demonstrate achievement of its intermediate outcomes pertaining to the use of CESI products by its target audiences, the initiative needs to monitor the extent to which CESI products are known, accessed, used and considered useful by its target users, and commit to regular reporting on the reach and use of the CESI products. It is recommended that the CESI DG Steering Committee oversee development and implementation of a performance measurement system for the CESI initiative. This system should include a database of target users of CESI products, a process for annually measuring user reach, satisfaction and use of CESI products, and a reporting strategy.

Recommendation 2: Several sources indicated that the target CESI users have limited awareness of the CESI products and website. Given the importance of increasing awareness of the CESI products, it is recommended that the CESI DG Steering Committee revisit the CESI communications strategy to identify and implement means to increase target CESI users’ awareness of CESI products.

Recommendation 3: In light of the significant resource investments made in the CESI initiative, it is important for the implicated federal departments to maximize the use and usefulness of the CESI products. All sources consulted for this evaluation indicate that the current improvements being made to the 2008 CESI products suite are likely to improve the usefulness of CESI products. Some sources, however, consider that the use of CESI for policy development and government decision making is hindered by the absence of a federal government commitment to specific and measurable objectives related to the CESI indicators.

a) It is recommended that once intended CESI improvements are completed, the CESI DG Steering Committee reassess the success of the CESI initiative.

b) It is recommended that the DG Steering Committee consider the role expected of CESI in supporting policy development and federal government decision making, and explore additional means to increase CESI’s usefulness such as possible linkages to the implementation of the 2008 Federal Sustainable Development Act.

7.0 MANAGEMENT RESPONSE

The Canadian Environmental Sustainability Indicators (CESI) initiative, on behalf of the Government of Canada, demonstrates Canada’s ability to tell the country’s sustainability story and report to Canadians on the state of the Canadian environment. For the past four years, CESI has demonstrated a record of organizing and disseminating information related to the environment in a scientifically credible and defensible way and tracking and reporting trends on air quality, water and greenhouse gas emissions.

Funding for the CESI initiative was initially approved in 2003–2004 for a period of five years. Since the funding sunsets in 2008–2009, Environment Canada, in partnership with Statistics Canada and Health Canada, is seeking opportunities for the renewal of the CESI initiative. All Management Response commitments to the Evaluation Recommendations will be dependent on the nature of the renewal of the initiative and the related available resources.

The Assistant Deputy Minister of the Strategic Policy Branch and the CESI DG Steering Committee Chair, Director General, Strategic Information Integration Directorate, accept the evaluation and its recommendations and have provided a preliminary plan to implement the recommendations within the context of a possible CESI renewal.

Recommendation 1: In order to demonstrate achievement of its intermediate outcomes pertaining to the use of CESI products by its target audiences, the initiative needs to monitor the extent to which CESI products are known, accessed, used and considered useful by its target users, and commit to regular reporting on the reach and use of the CESI products. It is recommended that the CESI DG Steering Committee oversee development and implementation of a performance measurement system for the CESI initiative. This system should include a database of target users of CESI products, a process for annually measuring user reach, satisfaction and use of CESI products, and a reporting strategy.

Response: The CESI DG Steering Committee agrees with Recommendation 1 to develop and implement a performance measurement system for the CESI initiative. In response, within three months of the renewal of CESI, the CESI DG Steering Committee will develop and present to the Internal Services (IS) Board, for its approval, a revised CESI Indicators Logic Model and a Performance Measurement Plan that will include clear accountabilities and timelines for implementation of the performance measurement plan and associated data collections mechanisms.

Performance measurement will include:

• An inventory of target users of CESI products
• A process for annually measuring user reach, satisfaction and use of CESI products, (monitoring client feedback on CESI products through focus groups/public opinion research and through web monitoring of client statistics which will provide valuable information to ensure that the CESI products meet federal government policy development and decision-making needs) and
• A reporting strategy

Recommendation 2: Several sources indicated that the target CESI users have limited awareness of the CESI products and website. Given the importance of increasing awareness of the CESI products, it is recommended that the CESI DG Steering Committee revisit the CESI communications strategy to oversee, identify and implement means to increase target CESI users’ awareness of CESI products.

Response: The CESI DG Steering Committee agrees with the recommendation to develop a more strategic and targeted marketing strategy. Within three months of the renewal of CESI, the CESI DG Steering Committee will present to the IS Board a CESI Engagement Strategy that will involve collaboration with the Department’s Communications Branch to develop and implement a strategy to identify the intended target audiences and promote awareness and use of the CESI report by Fall 2009.

This strategy would demonstrate the Government of Canada’s commitment to accountability and transparency to inform Canadians in an open and transparent manner about key environmental issues. The new Engagement Strategy will include: key principles and considerations, results of audience research, outreach objectives and proposed activities, monitoring and evaluation, budget and plan for implementation and campaign management. The strategy will address, in particular, engagement of interested partners to champion the dissemination and promotion of the CESI products.

Recommendation 3: In light of the significant resource investments made in the CESI initiative, it is important for the implicated federal departments to maximize the use and usefulness of the CESI products. All sources consulted for this evaluation indicate that the current improvements being made to the 2008 CESI products suite are likely to improve the usefulness of CESI products. Some sources, however, consider that the use of CESI for policy development and government decision making is hindered by the absence of a federal government commitment to specific and measurable objectives related to the CESI indicators.

a) It is recommended that once intended CESI improvements are completed, the CESI DG Steering Committee reassess the success of the CESI initiative.

Response: The CESI DG Steering Committee agrees with the recommendation to reassess the success of the CESI initiative. The CESI DG Steering Committee will initiate the intended improvements in the context of the renewal of CESI and will implement a cycle of evaluation of the initiative, aligned with the renewal time frame (i.e., mid term and final evaluation in a 5-year cycle). The CESI Indicators Logic Model and Performance Measurement Plan, developed in response to Recommendation 1, will be the basis for monitoring and tracking developments towards progress.

b) It is recommended that the DG Steering Committee consider the role expected of CESI in supporting policy development and federal government decision making, and explore additional means to increase CESI’s usefulness, such as possible linkages to the implementation of the 2008 Federal Sustainable Development Act (FSDA).

Response: The CESI DG Steering Committee agrees with the recommendation to consider the role expected of CESI in supporting policy development and federal government decision making. Beyond the specific reporting mandate of the initiative, the Strategic Information Integration Directorate will apply its experience with environmental indicator development to the creation of a departmental centre of expertise to provide guidance, policy advice and statistical analysis in support of the development of new indicators, as appropriate, to strengthen the information base for 1) department-wide activities such as the Departmental Performance Report or the Program Activity Architecture, as well as 2) government-wide activities such as the annual Canada’s Performance report and implementation of the 2008 Federal Sustainable Development Act (FSDA).

The revised CESI Indicators Logic Model, Performance Measurement Plan and data collection mechanisms, developed in response to Recommendation 1, will refine and clarify the intended target audiences. One audience is: federal government managers and staff with responsibility for the Federal Sustainable Development Strategy (FSDS). As a result, the CESI indicators will be the adopted indicators in the implementation of the 2008 Federal Sustainable Development Act by December 2009 in order to meet timelines laid out in the FSDA legislation.

Annex 1
CESI Logic Model

Logic Model for Indicators for a Competitive Economy and Sustainable Environment

Reach	Target Audience Members of the general public / Government managers and employees / decision makers / provinces and territories / municipalities	Ultimate Outcomes	“Canadians recognize and use the Indicators as one source of trusted information on air, water, and greenhouse gases.” Knowledge is shared effectively with target audiences and stakeholders are well served nationally Canadians better understand how their actions affect the quality of the environment and human health Political and senior decision makers are more informed and make better decisions
		Intermediate Outcomes	Practices Results are reported to Canadians Government implements actions in response to results Indicators tools / approaches used in policy / decision making
			Capacity Information system in place to support 3 indicators Monitoring systems are improved and expanded in areas of air, water, greenhouse gases Drill-down capacity (per site / local / regional / provincial) Ability to deliver information on a more timely basis (including monitoring)
	Supporting Sponsors / Partners Government Departments / environmental non-government organizations / technical experts / industry	Immediate Outcomes	Partner/Rules Support Identification of resource / information gaps leads to targeted partnerships and distribution of funds Provinces equipped to publish own environmental reports Coordinated national approach (federal, federal / provincial)
			Participation/Reaction Improved collaboration mechanisms are in place to ensure: distinct contributions complementary information systems coherent approach to provinces All provinces and federal departments are engaged: data provided/shared/evaluated Increased recognition for ongoing environmental monitoring programs
	Environment Canada/Health Canada/Statistics Canada	Activities and Outputs	Design and development of indicators Improvements to instrumentation for measuring and reporting on air quality and water quality Survey work to identify gaps in air quality data and to improve understanding of water quality available for major water uses Create a national source water quality index methodology Develop a method for linking air quality to human health Integrate the existing greenhouse gas emissions indicator into the framework for all indicators Publish print and web-based reports concerning the indicators

Annex 2
Key Secondary Sources Reviewed

Environment Canada. Strategic Information Integration Directorate (SIID), List of web sites, publications and events whereby CESI products were distributed, cited or used, 2007	This document shows tracking of the various products/events where CESI results were specifically cited or used in meeting, a Departmental obligation, and other promotional activities including events, online presence, workshops, media / newsletter / email distribution.
Environment Canada. Strategic Information Integration Directorate (SIID), CESI Review: Issues and Challenges 2007	Through an internal consultation process CESI staff and managers have tracked progress toward the achievement of deliverables. Achievements, issues and proposed solutions are presented in tabular format.
Environment Canada. Strategic Information Integration Directorate (SIID), Meeting summaries and records of decisions	Numerous meeting summaries and records of decisions demonstrate ongoing communication of roles, responsibilities and progress.
Environment Canada. Strategic Information Integration Directorate (SIID), CESI Highlights Questionnaire 2007	A questionnaire was distributed with the 2007 Highlights report. It aimed to measure the perceived usefulness of the 2006 and 2007 CESI publications. It was sent to 715 potential respondents, 17 addresses were no longer valid, 71 responded.
Statistics Canada Client Satisfaction Surveys 2006 and 2007	As per its mandate under the CESI initiative, Statistics Canada conducted satisfaction surveys in 2006 and 2007 to determine if the CESI reports have been successful. Survey candidates were those who had indicated an interest in participating in an evaluation of the electronic publications during a pre-solicitation for future research. In all, 64 candidates were contacted in May 2006 (11 responded) and 81 in 2007 (18 responded). The survey was posted on a website and was communicated by email. It asked about Satisfaction with the three CESI reports; Reasons for use; Most valued features; and Suggestions to improve content and presentation. Multiple choice answer questions used 5-point scales. Population, number of respondents and cell sizes were too small for the findings to be useful.
Phoenix Strategic Perspectives Inc., CESI User Needs Assessment	User needs assessment and product testing qualitative study conducted by an external consulting firm. The research included 4 focus groups with members of the general public with an interest in environmental issues, and 24 one-on-one interviews with stakeholders from other target audiences, including environmental non-governmental organizations and industry.
IISD, Review of the Canadian Environmental Sustainability Indicators Initiative	Study conducted by an external research group consisting of interviews with 13 high-level policy makers and experts in accountability reporting. The study addressed the following issues: To what extent has CESI achieved (or is on track to achieve) its expected immediate and intermediate outcomes for components for which Environment Canada is responsible or shares responsibility? Are the activities still appropriate for achieving CESI goals? If not, how might they need to change? Do the CESI activities benefit/leverage other Environment Canada / government goals or outcomes? Based on the CESI activities, products and analyses conducted so far and planned for the final year, what are the key gaps that would need to be filled to produce a more policy-relevant indicator suite? What are the institutional changes that are needed to reach this? Are there interim steps that can be taken? Furthermore, the scope was specified as: the activities, products, and outcomes related to CESI, but not expenditures; and the overall design and delivery of CESI, from the perspective of Environment Canada.

Annex 3
Complete List of Secondary Sources

Public Documents:

Canada. Canadian Environmental Sustainability Indicators 2006. Air Quality Indicators Data Sources and Methods Report. 2006.

Canada. Canadian Environmental Sustainability Indicators 2006. Feature Report. 2006.

Canada. Canadian Environmental Sustainability Indicators 2006. Freshwater Indicators Data Sources and Methods Report. 2006.

Canada. Canadian Environmental Sustainability Indicators 2006. Greenhouse Gas Indicators Data Sources and Methods Report. 2006.

Canada. Canadian Environmental Sustainability Indicators 2006. Highlights Report. 2006.

Canada. Canadian Environmental Sustainability Indicators 2007. Air Quality Indicators Data Sources and Methods Report. 2007.

Canada. Canadian Environmental Sustainability Indicators 2007. Feature Report. 2007.

Canada. Canadian Environmental Sustainability Indicators 2007. Freshwater Indicators Data Sources and Methods Report. 2007.

Canada. Canadian Environmental Sustainability Indicators 2007. Greenhouse Gas Indicators Data Sources and Methods Report. 2007.

Canada. Canadian Environmental Sustainability Indicators 2007. Highlights Report. 2007.

Canada. Department of Finance. Budget Plan 2004.

Canada. Department of Finance. Budget Speech 2000. February 28, 2000.

Canada. House of Commons Standing Committee on Environment and Sustainable Development. The Canadian Environmental Protection Act, 1999 - Five-Year Review: Closing the Gaps. April 2007. http://cmte.parl.gc.ca/cmte/CommitteePublication.aspx?SourceId=204099

Canada. Speech from the Throne to open the First Session of the Thirty-Eighth Parliament of Canada. October 5, 2004.

Canada. Treasury Board Secretariat. Canada’s Performance 2006: The Government of Canada’s Contribution. 2006.

Canada. Treasury Board Secretariat. Canada’s Performance 2007: The Government of Canada’s Contribution. 2007.

Canadian Information System for the Environment (CISE) website: http://www.cise-scie.ca/

National Round Table on the Environment and the Economy (NRTEE). The State of the Debate on the Environment and the Economy: Environment and Sustainable Development Indicators for Canada. 2003.

Pinter, Laszlo, Peter Hardi and Peter Bartelmus. Sustainable Development Indicators: Proposals for a Way Forward. Discussion Paper Prepared Under a Consulting Agreement on Behalf of the United Nations Division for Sustainable Development (New York, 13-15 December 2005). International Institute for Sustainable Development (IISD). 2005.

Sustaining the Environment and Resources for Canadians website: http://environmentandresources.gc.ca/default.asp?lang=En&nav=B3BE4E55-00

Task Force on a Canadian Information System for the Environment. Canadian Information System for the Environment: Sharing Environmental Decisions. October 2001.

Internal Documents:

Canadian Environmental Sustainability Indicators (CESI). CESI Governance Structure. July 4, 2007.

Canadian Environmental Sustainability Indicators (CESI). List of web sites, publications and events whereby CESI products were distributed, cited or used. September 27, 2007.

Environment Canada. CESI Logic Model. 2005.

Environment Canada. CESI Water and Air Quality Deliverables Status. January 26, 2007.

Environment Canada. CESI Water Quality Indicators Mid-Term Progress Report for 2007. Draft for Discussion. Version 4. June 27, 2007.

Environment Canada. Communications Advisory note: Publication Release. The Canadian Environmental Sustainability Indicators (CESI) Report 2008–2009.

Environment Canada. Communications Advisory note: Publication Release. The Canadian Environmental Sustainability Indicators (CESI) Report 2007–2008.

Environment Canada. Communications Advisory note: Publication Release. The Canadian Environmental Sustainability Indicators (CESI) Report 2006–2007.

Environment Canada. Engagement Strategy: Canadian Environmental Sustainability Indicators Initiative 2007-2008.

Environment Canada. Strategic Information Integration Directorate (SIID). Canadian Environmental Sustainability Indicators (CESI): Where Are We now? (PowerPoint Deck) July 17, 2007.

Environment Canada. Strategic Information Integration Directorate (SIID). Environmental Sustainability Indicators – Modernizing the State of the Environment Reporting (PowerPoint Deck for Strategic Integration Board). February 14, 2008.

Environment Canada. Strategic Information Integration Directorate (SIID). CESI Highlights Targeted Distribution (December 13, 2007) – Questionnaire Findings. 2007.

Environment Canada. Strategic Information Integration Directorate (SIID). Next Generation. State of Environment Reporting (PowerPoint Deck). March 27, 2008.

Environment Canada. Strategic Information Integration Directorate (SIID). CESI Review: Issues and Challenge. January 26, 2007.

Environment Canada. Strategic Information Integration Directorate (SIID). Review of CESI 2005: Areas for Discussion.

Environment Canada. Strategic Information Integration Directorate (SIID). How EC Works On CESI. Items for Discussion, EC Indicator Team Leads Meeting. January 31, 2006.

International Institute for Sustainable Development (IISD). Review of the Canadian Environmental Sustainability Indicators Initiative. Final Report. April 3, 2008.

Phoenix Strategic Perspectives Inc. Final Report. Canadian Environmental Sustainability Indicators Initiative. User Needs Assessment and Product Testing. December 2007.

Statistics Canada. 2007 Evaluation: Canadian Environmental Sustainability Indicators Reports – Preliminary Tabular Results (May 30, 2007–June 7, 2007). June 8, 2007.

Statistics Canada. Benchmarking of Statistics Canada CESI Performance Against Commitments. January 26, 2007.

Statistics Canada. Draft Report: Tabular Results for 2006 Environment Client Satisfaction Assessment Survey for CESI 2005. Canadian Environmental Sustainability Indicators Reports. May 2006.

Statistics Canada. Suggestions to Improve the CESI Report and Development Process for 2007. January 18, 2007.

Annex 4
Key Informant Interview Guide

The Audit and Evaluation Branch of Environment Canada is conducting an evaluation of the Canadian Environmental Sustainability Indicators (CESI) initiative. The purpose of this evaluation is to assess the relevance, success, cost effectiveness and design and delivery of the initiative since its inception in 2005 in order to meet Treasury Board requirements. A first phase of this evaluation was completed this spring and consisted of a review of existing documentation.

As part of the second phase of this research, we are conducting key informant interviews with managers involved in CESI’s implementation in order to obtain your perspectives on the issues being examined.

The information collected as part of these interviews will be held in strict confidence and will not be attributed to specific individuals; rather, responses will be rolled up into a higher level assessment of the initiative. The interview is expected to take between 45–60 minutes.

---

1. Could you briefly describe your role and responsibilities with respect to the CESI initiative?

2. I’m going to identify a number of outcomes identified for the CESI initiative, and for each one I’d like to know to what extent you think that CESI is achieving each particular outcome, and why you say that?

• Reporting results to Canadians
• Government implementing actions in response to results
• Indicators tools/approaches being used in policy/decision making (by governments and/or the general public)
• Information systems in place to support the three indicators
• Monitoring systems improved and expanded in areas of air, water, GREENHOUSE GAS EMISSIONS
• Drill-down capacity (per site/local/regional/provincial)
• Ability to deliver information on a more timely basis (including monitoring)
• Identification of resources/information gaps leading to targeted partnerships and distribution of funds
• Provinces being equipped to publish their own environmental reports
• Coordinated national approach (federal, federal/provincial)
• Improved collaboration mechanisms in place

3. Recent CESI senior management briefings have highlighted current plans for CESI to transition to a new State of the Environment Reporting vision. Are you aware of this new vision? If so, could you please explain what it consists of?

4. As part of senior management briefings, it was suggested that improvements were being made to the 2008 CESI products in order to increase their usefulness and accessibility. Were you aware of this?

5. Could you please describe what these intended changes entail?

6. I’m now going to identify some perceived gaps or weaknesses in CESI identified by users among the general public and federal decision makers. Using a 5-point scale, where 1 means not effective at all and 5 means very effective, I’d like to know to what extent you think recent changes in the last year or planned changes will be effective in addressing them.

• Availability of data at more local scales/drill-down capacity (e.g., at the site, local, regional and provincial levels)
• Better linkages to issues of interest to the general public and Members of Parliament (e.g., human health issues, linkage with socio-economic indicators)
• Increased usefulness of CESI indicators in accountability reporting (i.e., to assist the federal government in reporting on the achievement of specific goals or policy objectives)
• Need for better navigability of the website
• Need for increased awareness of CESI products by the general public
• Need for official government indicators that would serve as THE source of official data from the federal government.

I’d now like to explore issues related to CESI’s governance and accountability framework. In order to help guide the discussion, please refer to the organization chart presenting the CESI governance structure, appended to the interview guide.

7. In your opinion, to what extent is the governance of the CESI initiative effective? Please explain.

8. In your opinion, are there any (other) changes that should be made to the CESI initiative that would have a direct impact on improving its efficiency or effectiveness? If so, what?

9. To your knowledge, do other sources of indicators and data related to Canada’s environmental sustainability currently exist? If so, what are they? How is duplication avoided and complementarity achieved between CESI and this/these other indicator(s)?

10. How do you plan to measure user satisfaction and usefulness of the CESI products in the future?

11. Do you have any other observations to add regarding CESI’s relevance, success, or effectiveness?

12. Should the need arise, would it be possible to contact you to revisit any of the issues explored during the interview? Should we feel the need to do this, we would only take a few additional minutes of your time.

THANK YOU VERY MUCH FOR YOUR TIME AND THOUGHTFUL FEEDBACK.
IT IS VERY MUCH APPRECIATED.

Annex 5
Summary of Findings

Evaluation Question (EQ)	Achieved	Progress Made, Attention Needed	Little Progress, Priority for Attention	Not Applicable
Relevance:
EQ1 Role of government in CESI	X
EQ2 Connection with societal and environmental needs	X
Success:
EQ3 Achievement of outcomes		X
EQ4 Unintended outcomes				X
Cost Effectiveness:
EQ5 Avoidance of duplication and achievement of complementarity	X
EQ6 Increasing efficiency	X
Design and Delivery:
EQ7 Effectiveness of CESI governance	X
EQ8 Performance measurement collection and use in decision making		X
EQ9 Clear deliverables and expected results and program delivered as designed		X

[1] International Institute for Sustainable Development (IISD). Review of the Canadian Environmental Sustainability Indicators Initiative. Final Report. April 3, 2008.

[2] Task Force on a Canadian Information System for the Environment. Canadian Information System for the Environment: Sharing Environmental Decisions, October 2001, pp. 9–13.

[3] International Institute for Sustainable Development (IISD). Review of the Canadian Environmental Sustainability Indicators Initiative. Final Report. April 3, 2008. p. 4.

[4] Ibid, p. 5.

[5] http://environmentandresources.ca/default.asp?lang=En&n=2102636F-1

[6] The 2008 CESI product suite includes modified features that will be described later in the findings section of this report. Since this new suite has not yet been published, the description provided herein refers to the 2007 CESI product suite.

[7] For the purpose of this study, managers include directors general, directors, chiefs and managers.

[8] Canadian Environmental Protection Act, 1999. Section 44(1)(a).

[9] Task Force on a Canadian Information System for the Environment. Canadian Information System for the Environment: Sharing Environmental Decisions. October 2001. p. 14.

[10] Canada. Department of Finance. Budget Speech 2000. February 28, 2000. pp. 14–15.

[11] Canada. Department of Finance. Budget Plan 2004. pp. 142, 161.

[12] Phoenix Strategic Perspectives Inc. Final Report. Canadian Environmental Sustainability Indicators Initiative. User Needs Assessment and Product Testing. December 2007. p. 25.

[13] Task Force on a Canadian Information System for the Environment. Canadian Information System for the Environment: Sharing Environmental Decisions. October 2001. pp. 4–6.

[14] National Round Table on the Environment and the Economy (NRTEE). The State of the Debate on the Environment and the Economy: Environment and Sustainable Development Indicators for Canada. 2003. pp. xviii, 15.

[15] Canadian Information System for the Environment (CISE) website: http://www.cise-scie.ca/.

[16] Environment Canada. Strategic Information Integration Directorate (SIID). CESI Highlights Targeted Distribution (December 13, 2007) – Questionnaire Findings. 2007.

[17] Phoenix Strategic Perspectives Inc. Final Report. Canadian Environmental Sustainability Indicators Initiative. User Needs Assessment and Product Testing. December 2007. pp. 6, 9–10.

[18] International Institute for Sustainable Development (IISD). Review of the Canadian Environmental Sustainability Indicators Initiative. Final Report. April 3, 2008. p. 24.

[19] Canada. House of Commons Standing Committee on Environment and Sustainable Development. The Canadian Environmental Protection Act, 1999 - Five-Year Review: Closing the Gaps. April 2007. http://cmte.parl.gc.ca/cmte/CommitteePublication.aspx?SourceId=204099

[20] Phoenix Strategic Perspectives Inc. Final Report. Canadian Environmental Sustainability Indicators Initiative. User Needs Assessment and Product Testing. December 2007. p. 9.

[21] Ibid., pp. 28–33.

[22] Environment Canada. Strategic Information Integration Directorate (SIID). CESI Highlights Targeted Distribution (December 13, 2007) – Questionnaire Findings. 2007.

[23] Phoenix Strategic Perspectives Inc., pp. 25, 34.

[24] Environment Canada. Strategic Information Integration Directorate (SIID). Next Generation. State of Environment Reporting (PowerPoint Deck). March 27, 2008; CESI key informants.

[25] International Institute for Sustainable Development (IISD). Review of the Canadian Environmental Sustainability Indicators Initiative. Final Report. April 3, 2008. p. 27.

[26] Ibid, p. 22.

[27] Phoenix Strategic Perspectives Inc. Final Report. Canadian Environmental Sustainability Indicators Initiative. User Needs Assessment and Product Testing. December 2007. pp. 30–31.

[28] International Institute for Sustainable Development (IISD). Review of the Canadian Environmental Sustainability Indicators Initiative. Final Report. April 3, 2008. p. 22.

[29] Environment Canada. Strategic Information Integration Directorate (SIID). Next Generation. State of Environment Reporting (PowerPoint Deck). March 27, 2008.

[30] Environment Canada. Strategic Information Integration Directorate (SIID). CESI Review: Issues and Challenge. January 26, 2007.

[31] Phoenix Strategic Perspectives Inc. Final Report. Canadian Environmental Sustainability Indicators Initiative. User Needs Assessment and Product Testing. December 2007. p. 25.

[32] International Institute for Sustainable Development (IISD). Review of the Canadian Environmental Sustainability Indicators Initiative. Final Report. April 3, 2008. p. 26.

[33] Environment Canada. Strategic Information Integration Directorate (SIID). CESI Review: Issues and Challenge. January 26, 2007.

[34] Canadian Environmental Sustainability Indicators (CESI). CESI Governance Structure. July 4, 2007.

[35] Environment Canada. Strategic Information Integration Directorate (SIID). CESI Review: Issues and Challenge. January 26, 2007.

[36] International Institute for Sustainable Development (IISD). Review of the Canadian Environmental Sustainability Indicators Initiative. Final Report. April 3, 2008. p. 27.