Home > Proactive Disclosure > Audits and Evaluations > A&E Reports > 2010-2011 > Evaluation Review of the Clean Air Agenda Adaptation Theme: Review of Program Evaluation and Performance Measurement Findings

Evaluation Review of the Clean Air Agenda Adaptation Theme: Review of Program Evaluation and Performance Measurement Findings

ToC | Next page

Executive Summary

Introduction

The purpose of this document is to present a review of the available evaluation findings concerning the Adaptation Theme programs under the Clean Air Agenda (CAA). The Adaptation Theme Evaluation Review is a synthesis of the current findings of individual evaluations and available performance measurement data for the seven programs within the Adaptation Theme. These evaluations were conducted, or are currently being conducted, by their respective responsible departments and agency. The Adaptation Theme Evaluation Review will also be integrated into the overall Clean Air Agenda Horizontal Evaluation Report for the fall of 2010.

All program-level evaluations sought to address program-specific issues of relevance and performance and used a core set of evaluation questions in order to facilitate the review of findings at the theme level.

Methodology

Two methods were used in the Adaptation Evaluation Review:

• Individual program evaluations – The majority of the evidence presented in the Adaptation Evaluation Review is based on the available findings, conclusions and recommendations from the individual program evaluations. The individual evaluations generated findings based on multiple lines of evidence, such that the evidence reviewed here already represents numerous sources of evidence.
• Adaptation Theme Evaluation Questions – eight program managers in participating departments were asked three additional “theme-specific” questions. These questions, developed in consultation with evaluation representatives from all participating departments, collected qualitative evidence to assess the effectiveness and challenges of the Adaptation Theme’s management and accountability approach.

Top of Page

The following two limitations to the Adaptation Theme Evaluation were also noted:

• The programs under the Adaptation Theme were new programs that had been operational for roughly 18 months to two years at the time of the individual program evaluations. As noted in many of the individual evaluations, the Adaptation Theme programs generally did not begin to reach external stakeholders until late 2008-2009 and early 2009-2010. As was anticipated during the evaluation planning phase, the individual evaluations generally provided evidence related to outputs and a limited assessment of early outcomes, given the early stage of program implementation. The findings in the Adaptation Evaluation Review were therefore based on early assessments of program impacts; and,
• Given the short implementation period for Adaptation activities, many of the program evaluations were expected to be completed late in the 2010-2011 fiscal year. The following thematic evaluation findings are, as a result, a combination of early evaluation results and performance information. Only two evaluations were conducted and approved by their respective Departmental Evaluation Committees. Two other evaluations had been completed but were not yet approved; two evaluations were under way and were able to provide preliminary evaluation data; and, one evaluation had not yet begun, though the program was able to provide results from a stakeholder survey. Readers should be cautioned that findings and conclusions presented in the report represented the evidence available at a specific point in time (spring, 2010).

Top of Page

Findings/Conclusions

Since the evaluation findings are based on evidence from programs early in implementation, the conclusions are limited in scope:

• The Adaptation Theme appears to be a relevant government activity based on the results of the Adaptation Evaluation Review. The programs funded under the Adaptation Theme addressed areas of federal government responsibility and address on-going climate change and air quality issues that would not be otherwise addressed; and
• The evidence, both financial and non-financial, concerning expected implementation, as well as the positive comments made by Theme stakeholders on the overall management and accountability of the Adaptation Theme, appears to indicate that the programs within the Adaptation Theme have taken the appropriate preliminary steps to implement the Theme. The outcome evidence that is available also points to the programs moving towards the achievement of the goals of the Theme, albeit at a very early stage of development. Further, though stakeholders suggested that the Adaptation Theme was less cohesive than other Clean Air Agenda Themes at its outset, the Theme management structure, particularly the Directors General Management Committee, appears to be operating effectively. Evidence on early best practices noted the importance of engaging with external stakeholders and recognizing that adaptation is one of many competing priorities and will take sustained effort on the part of all partners to address the issues. Given the early indicators of success, and the statements made by Adaptation Theme stakeholders regarding the uncertain future of the Theme beyond March 2011, there is a need for additional formal planning as a Theme to decide how to move forward based on the preliminary evidence of success.

Recommendation

The following recommendation was made to responsible departmental Assistant Deputy Ministers. Given the early evidence that the Adaptation Theme is a relevant issue for government, and that the Theme’s activities appear to be on track toward achieving the desired outcome, balanced against the uncertain future of adaptation funding and the fact that the Theme was developed quickly and was less cohesive than desired, it is recommended that the responsible Adaptation Theme Assistant Deputy Ministers and other Assistant Deputy Ministers, as appropriate, articulate a plan for possible future adaptation programming and communicate this plan to the responsible Deputy Ministers.

Top of Page

Management Response

Statement of Agreement/Disagreement with the Recommendation

The ADMs responsible for the adaptation programs that received funding for the 2007-2011 period agree with the recommendation. These ADMs include: Strategic Policy Branch, EC; Science and Technology Branch, EC; Meteorological Service of Canada, EC; Earth Sciences Sector, NRCan; Northern Affairs, INAC; Healthy Environments and Consumer Safety Branch, HC; First Nations and Inuit Health Branch, HC; and Infectious Disease and Emergency Preparedness Branch, PHAC.

Management Action

All existing adaptation programs will sunset on March 31, 2011. As such, the relevant ADMs are currently undertaking work to address this recommendation as part of the broader Clean Air Agenda renewal.

The ADMs responsible for adaptation policy and programming are developing a consensus-based interdepartmental plan to inform future government decisions for the Adaptation Theme. This plan will be presented to departments participating in the Adaptation Theme renewal.

The responsible ADMs include: Strategic Policy Branch, EC; Science and Technology Branch, EC; Meteorological Service of Canada, EC; Science and Policy Integration, NRCan; Earth Sciences Sector, NRCan; Northern Affairs, INAC; Healthy Environments and Consumer Safety Branch, HC; First Nations and Inuit Health Branch, HC; Infectious Disease and Emergency Preparedness Branch, PHAC; Policy Development, TC; Agri-Environment Services Branch, AAFC; Strategic Policy, DFO; Policy and Stakeholder Relations Branch, SCC; the National Parks Directorate, PCA; and Global Issues Branch, DFAIT.

A Directors General Management Committee (DGMC) coordinates all Theme activities on behalf of the Adaptation Theme ADMs, and is crucial to the overall functioning of the Theme. While the program lead of the DGMC rotates between departments, Environment Canada is the policy lead for climate change adaptation and is therefore responsible for the policy authority of the theme, including the implementation of management actions.

ToC | Next page