Final Report
November 2010
Previous Page | Table of Contents | Next Page
This evaluation assessed the relevance and performance of the Improved Climate Change Scenarios Program from 2007–2008 to 2009–2010 to provide information to senior management for decision making and to provide results for the Adaptation Theme evaluation to be rolled-up to the CAA-level in the fall of 2010. Five key issues were examined, as presented in the Evaluation Matrix in Annex 2:
Research and associated program activities that were to be resourced through the incremental funding for the Improved Climate Change Scenarios Program under the Adaptation Theme were assessed as part of this evaluation. Data were collected from August through October 2009, with some follow-up interviews conducted in November 2009 to capture the impact of organizational changes within ASTD that occurred after the first phase of the evaluation had already been completed. Additional documents were requested throughout the analysis and reporting phases of the evaluation to fill gaps in data and to capture information on the Program's ongoing progress towards the achievement of expected outcomes. Because the implementation of the Improved Climate Change Scenarios Program was in its early stages at the time of data collection and analyses, assessment of the Program's performance focused on activities, outputs and immediate outcomes that were measurable at that time. Longer-term outcomes (intermediate and final outcomes) were examined for demonstration of progress towards their achievement.
Data were collected through a review of documents and key informant interviews to address all evaluation issues described in the Evaluation Matrix in Annex 2.
Document Review – The Program was requested to provide program documents to demonstrate its relevance and performance. The evaluation team reviewed key documents including Speeches from the Throne, federal budgets, Reports on Plans and Priorities (RPPs), Departmental Performance Reports (DPRs), peer reviewed research papers and presentations, planning documents, commissioned reports, documents related to stakeholder consultations and meetings, financial information and quantitative data from participant evaluations of training sessions. (A complete list of documents reviewed is presented in Annex 3.) Evaluators reviewed these documents and compiled data in a source document which was then analyzed in order to address each evaluation question.
Key Informant Interviews – A total of 50 key stakeholders were interviewed from the following groups:
All names and contact information for interviewees were provided by the Program. Semi-structured interviews were conducted in the interviewee's language of choice, using an interview guide tailored for each category of key informant. Interviews were conducted either in person or by phone.
There were some limitations and challenges to this evaluation.
Implementation of the Program is in its early stages
This evaluation was scheduled to take place in the third year of the Program's implementation in order for results to be available in time for a roll-up to the thematic and CAA levels. Due to funding issues involving CCIS and delays in receiving Adaptation Theme resources, however, program activities could not fully begin until approximately one year prior to the launch of this evaluation. In effect, the Program was in place for less than one year when the evaluation began. As such, evidence was not available to assess progress toward all intended program outcomes.
Re-structuring of program occurred mid-evaluation
Changes in the organizational structure of the Program occurred midway through the evaluation. The Adaptation and Impacts Research Section (AIRS), previously Adaptation and Impacts Research Division (AIRD), was subsumed under the Climate Research Division (CRD). Because AIRD was a division, it operated under its own budget and reported directly to the Director General of ASTD. Because this change occurred after the data collection for the evaluation was completed, follow-up interviews were conducted with key program managers and senior managers to obtain evidence on the potential impacts of this change. Most of the data collected through interviews, however, reflect the Program prior to this change in organization.
Attribution of expected outcomes to program is difficult
It was difficult to connect the achievement of expected outcomes to Improved Climate Change Scenarios activities due to three confounding factors. First, climate research has a long history in the Department, making it difficult to attribute the achievement of outcomes directly to program activities and outputs that occurred within the time period examined through the evaluation. Second, the incremental funding for the Improved Climate Change Scenarios Program obtained under the Adaptation Theme was intended to support additional research and its related activities, particularly staffing actions. As a result, it was difficult to attribute the achievement of expected outcomes solely to those activities funded under this theme. Third, the research produced as part of the Improved Climate Change Scenarios Program necessarily requires an iterative process that involves input from multiple sources to test findings and develop solid conclusions. As such, the achievement of expected outcomes relies not only on the Improved Climate Change Scenarios Program but on research activities from the wider scientific community upon which the Improved Climate Change Scenarios Program relies.
Because of this difficulty in attribution, emphasis was placed on the contribution of activities and outputs to the achievement of expected outcomes rather than on attributing observed results to the Program.