Final Report
November 2010
The evaluation of the Improved Climate Change Scenarios Program was conducted by the Evaluation Division of Environment Canada's Audit and Evaluation Branch (AEB) in 2009‑2010 to meet the information requirements of senior management and to contribute results to the thematic and Clean Air Agenda (CAA) level evaluations by the fall of 2010. This evaluation assessed the relevance and performance of the Program from 2007‑2008 to 2009‑2010, examining five key issues:
The Improved Climate Change Scenarios Program is one of seven programs that together comprise the Adaptation Theme of the Clean Air Agenda (CAA). The CAA is a horizontal federal strategy that represents a part of the federal government's broader efforts to address the challenges of climate change and air pollution, with a view to building a clean and healthy environment for Canadians. As part of the Adaptation Theme, the Improved Climate Change Scenarios Program is designed to facilitate changes in the behaviour of Canadians to develop and use knowledge, information, tools and collaborative arrangements in order to take appropriate actions to reduce their risks from the impacts of climate change. More specifically, the objective of the Improved Climate Change Scenarios Program is to contribute to the availability and use of improved information on the future climate, climate extremes and climate change impacts.
The Program's activities focus on three key components: the enhancement and operation of global and regional climate models; the development of climate scenarios for climate, weather extremes and hazards; and the development of specialized information on hazards and climate extremes for infrastructure design. The first program area, climate modelling, is conducted by scientists at the Canadian Centre for Climate Modelling and Analysis (CCCma) while the other two program areas, climate change scenarios and development of information on hazards and climate extremes, are carried out by scientists in the Adaptation and Impacts Research Section (AIRS). Climate models are computer generated simulations that allow scientists to estimate climate conditions globally and regionally as well as to simulate climate change. Using these climate models, climate change scenarios define a range of possible future climate conditions to assess future hazards and extreme climate conditions that could develop as a result of climate change. This information feeds into the development of tools and information related to climate extremes that assist in disaster management planning and updating Canada's infrastructure codes and standards to include impacts of climate change.
Data for the evaluation were collected through a review of documents and key informant interviews. Because the implementation of the Program was in its early stages at the time of data collection and analyses, assessment of the Program's performance focused on activities, outputs and immediate outcomes that were measureable at that time. Intermediate and final outcomes were examined for demonstration of progress towards achievement.
Evaluation findings are summarized in the following sections according to evaluation issue.
Program activities within the Improved Climate Change Scenarios Program meet climate change information needs by contributing foundational data and information for decision making. The research and related activities conducted by the CCCma, the Canadian Climate Change Scenarios Network (CCCSN) and the hazards and extremes component contribute to an understanding of climate change and its present and future impacts. Without this research, municipal and provincial partners as well as professionals in industry and those dealing with national codes and standards would have greater difficulty making informed, strategic decisions to reduce the vulnerability of Canadians to climate change impacts and extreme weather. Evaluation findings indicate that more data and information is required, however, particularly on Canada's North as well as updated information that feeds into infrastructure design.
The Program's objectives are aligned with federal government priorities through their contributions to climate change science and adaptation and impacts research. These program activities support federal commitments to address climate change issues both in Canada and internationally. As well, the objectives of the Improved Climate Change Scenarios Program are strongly aligned with the priorities of Environment Canada. Program activities in climate modelling are linked to departmental priorities to improve the knowledge and information on weather and environmental conditions to influence decision making. In addition, research on adaptation and impacts is linked to Environment Canada's priorities to ensure that Canadians are informed of and respond appropriately to current and predicted environmental conditions.
There is a legitimate and necessary role and responsibility for the federal government to carry out the Program. Because climate change issues are broad across regions, departments, jurisdictions and industrial sectors, there is a need for a central organization to offer consistent and quality information. It is appropriate for the federal government to fulfil this role because it is able to provide the resources, infrastructure and capacity to mobilize the scientific community and coordinate efforts for improved knowledge on climate change impacts. Although there are other programs and organizations that provide similar information and data, these are complementary to the Improved Climate Change Scenarios Program and, because of its national focus, the Program does not duplicate the activities of these other programs and organizations. Without the Program, there would be gaps in information and data on climate change impacts and climate extremes as well as an absence of national standards for infrastructure, which could ultimately expose Canadians to increased risks to their safety and security.
Despite experiencing some delays in funding and challenges in the staffing process, the Program is achieving its intended outputs and immediate outcomes. Findings indicate that the Program has also made progress in achieving its intended intermediate and long-term outcomes, although it is still too early in program implementation to observe their full achievement.
The Improved Climate Change Scenarios Program is achieving all of its intended outputs. Work has been completed by CCCma on improvements to the Canadian global climate model and on the development of a regional climate model. There is also evidence of the development of tools for climate change scenarios as well as updates and improvements to the Hazards websites. All three program areas have been actively involved in producing research as well as contributing to the international scientific community through their involvement in the Intergovernmental Panel on Climate Change (IPCC). There is also strong evidence that the Program has engaged partners and established or expanded their networks. However, evidence from both interviews and documents demonstrate that the Program may face challenges in meeting future data and information demands because they are limited by internal capacity. Although the demand for CCCSN hands-on training is high, the Program is limited by the number of program staff available to conduct these sessions. As well, because capacity is limited for the hazards and extremes component of the Program, it has not been able to deliver on all of its commitments.
Evidence from interviews and document review demonstrates that the Program's partnerships and networks have contributed to increased collaboration to address climate change issues, both in Canada and internationally. The Program's involvement in international organizations facilitated collaborative research with partners in the international scientific community and the availability and access to research and program tools as well as the knowledge and expertise provided by program officers contributed to a higher level of collaboration among external partners.
The Program has also shown progress in contributing to the increased awareness of, availability of and access to climate information through its websites, through CCCSN training sessions and through information sessions with key stakeholders on hazards and climate extremes. The increasing number of data downloads and requests from the Program's websites from 2007 to 2009 by target audiences demonstrate increasing awareness of climate information. Positive feedback from target audiences who participated in CCCSN's training sessions and the hazards and extremes information sessions indicate that they have increased their awareness of and access to climate information. However, they expressed concern that the Program has limited capacity in delivering these products.
The Program has shown some progress towards achieving its expected intermediate outcomes but more needs to be done. Data from interviews suggest that some of the Program's target populations may not have sufficient scientific expertise to use the climate information available on program websites without the assistance of program staff. As well, evidence indicates that users of information find regionally or locally focussed information more useful to their work than national or global information and updated climate change data and information on climate extremes are required to support changes to codes and standards for infrastructure.
Evidence indicates that there is some progress in the application of research and climate information among user groups to assess climate change risks and to plan for future climate extremes but more needs to be done, particularly in incorporating climate change risks into infrastructure designs. Overall, evidence suggests that it is too early to observe stakeholders adapting their behaviour to reduce vulnerabilities to the impacts of climate change. Similarly, it is too early to observe the reduction of risks to communities, infrastructure and the health and safety of Canadians.
Evidence suggests that the Program is generally on track to being implemented as planned but it is not clear whether or not complete implementation will take place within the Program time frame given delays in delivering on program commitments. CCCma's work on climate modelling was delivered as planned and, while most activities were on track for CCCSN and the hazards component, CCCSN experienced delays in developing new tools for developing climate change scenarios and the hazards component was late in developing new methodologies for extreme event analyses. Data suggest that this was partly due to delays in funding and staffing issues. Because the Program is science-based and requires FTEs to produce its outputs, delays in funding had a negative affect on the Program's ability to hire staff as planned. This difficulty in hiring scientists was compounded by the limited supply of qualified candidates and by delays in departmental staffing actions, particularly for the hazards and climate extremes component that had the largest number of positions to staff. AIRS was able to hire scientists on a contract basis but these contracts are set to end in March 2011 when Adaptation funding ends.
As well, the Program's performance measures were neither comprehensive nor appropriate overall. The performance measurement plan that is available is not fully populated, and the performance indicators provided are difficult to measure and not specific enough to provide an adequate assessment of the Program's performance. Some performance data are available (e.g., performance data on website use and participant evaluations from training workshops) but other than this information, there is little evidence that the Program is collecting ongoing performance data. The Program's briefing documents to senior management provided information on progress towards planned activities or outputs, but it was difficult piecing together the full performance story because documents were not consistent across fiscal years for each program component. Part of the challenge of providing useable data for reporting and evaluation purposes may be in converting the Program's highly technical and scientific activities and outputs into a form that is easily understandable.
A few areas for improvement in program delivery were identified by interviewees. The reach of CCCSN training sessions could be expanded given that the number of people who receive training is small compared to the number of professionals who need to be able to use the Program's data and information. For those without the benefit of these training sessions, the websites are difficult to navigate and interviewees felt that it would be beneficial to have a more user-friendly interface as well as ongoing access to the scientific expertise of program staff to answer questions.
Evaluation evidence suggests the Program is employing cost-efficient measures. For example, the Program's strategy of building in-house science capacity facilitates knowledge transfer from senior-level scientists and helps the Program to achieve its goals and expected activities at the lowest possible cost. As well, synergies are created between these program scientists and various research networks in universities and regional research groups. This enhances the internal research capacity within the Department while reducing the duplication of research and saving on some overhead costs. CCCSN training sessions are also efficient in that they reach many members of the research community at once rather than having to address individual questions by users. However, a cost-efficiency analysis was not possible because of insufficient financial information.
The economy of the Program was likewise difficult to assess. A full cost analysis comparing resources expended to the achievement of outcomes was not possible because it is too early in program implementation to observe some of the Program's outcomes and there was a lack of clear, detailed documents outlining all program outputs and activities, their associated costs, and progress towards the achievement of intended outcomes.
The following conclusions on the Program's relevance and performance have been developed based on evaluation findings and analyses:
The Improved Climate Change Scenarios Program is relevant in meeting climate change information needs and is aligned with governmental priorities and federal roles and responsibilities.
The Program is fundamental in its provision of foundational data and information on climate change and its impacts. Because the Program's objectives and activities provide specialized scientific research on climate change and its impacts, its relevance is linked to the Government of Canada's broader policy objectives and priorities related to climate change issues. Program activities, outputs and intended outcomes are aligned with Environment Canada's priorities to improve knowledge and information on weather and environmental conditions to influence decision-making. These data and information help to influence changes to infrastructure design that take into account present and future climate hazards and extremes, ultimately contributing to protect the safety and security of Canadians and their property.
There is a legitimate role and responsibility for the federal government to deliver this program because, without these data and information, Canada would be more vulnerable to climate change and its impacts that pose considerable threat to the health and well-being of Canadians. This evaluation demonstrates that the federal government has a necessary role and responsibility in carrying out climate modelling, establishing and operating climate monitoring networks, conducting impacts and adaptation research in response to policy needs and providing national leadership in the direction, co-ordination, and funding of climate science in Canada. The Government of Canada is well suited for this role because it is able to provide the large amount of scientific expertise, resources, infrastructure and capacity (including technical capacity) needed to address climate change impacts for the long-term. Other programs and organizations involved in similar activities are complementary rather than duplicative the Program. As well, there is a need for centralization to provide consistent information and quality data with an independent, national perspective that can contribute to the international scientific community. Because climate change is a global phenomenon that must be addressed in concert with other countries, the Program's research activities in the international scientific community contribute to shared knowledge and understanding of climate change and its impacts around the world.
Current resource levels and difficulties in staffing pose a challenge for the Program in achieving its expected outcomes, particularly for the hazards and extremes component of the program.
There is a clear tension between expectations for the Program to achieve its intended outcomes and its ability to deliver on its commitments fully. The Improved Climate Change Scenarios Program is a science-based initiative that relies on the expertise of its scientists to meet its research objectives and achieve its expected outcomes. Adaptation funding for this program was intended to provide salary dollars for new work on climate modelling, climate change scenarios and, in particular, hazards and extremes research. Because there were delays in receiving Adaptation funding, the Program was unable to staff positions as planned to complete the work required. Departmental funding adjustments during the second year of implementation also had a negative impact on the Program's ability to achieve its intended outcomes. For example, there were delays in completing research on settlements in the North and AIRS was not able to complete on time its extreme event analyses to support the development of new national codes and standards. The activities and intended outcomes are not considered to be realistic as originally planned given the Program's capacity, particularly for the hazards and extremes component, which required the highest number of positions to be staffed.
The Program could be improved by expanding its reach and providing more useful information to target audiences.
One of the Program's strengths relating to cost-efficiencies are the synergies created with research networks in universities, regional research groups and regional Environment Canada staff in order to harness scientific expertise to achieve program objectives. These networks and partnerships have reduced duplication of effort, saved on overhead costs andhave led to greater collaboration to address climate change issues. However, improvements could be made to the kinds of partnerships and networks possible. For example, increased partnerships with regional organizations could lead to more joint research projects and stronger connections with other federal departments could increase communication and coordination of climate change research activities across the federal government. As well, CCCSN training workshops and information sessions for the hazards component of the Program were generally well-received by target audiences, but key informants felt that these reach only a portion of their potential target users. By expanding the reach of these sessions, awareness could be raised among more target users of the Program's products (e.g., websites, research) and could facilitate their use and application of the Program's climate change data and information. Expanding the Program's partnerships and networks through improved outreach activities could enhance its ability to achieve its expected outcomes.
As well, the Program could be improved by providing more useful information. Climate change impacts are not experienced on a global scale; extreme weather events are experienced locally and adaptive strategies therefore need to be locally focused to be effective. In particular, regionally or locally focussed information is more useful than national or global information for external partners. The general consensus across internal and external partners, however, is that regionally focussed climate change information is generally not sufficiently available, particularly for the North given the greater rate of climate change in this area and the impact for northern communities. As well, technical committees for developing infrastructure among external partner require updated data and information on climatic design values to support the development of new national codes and standards. Without these updated data and information, Canadians are increasingly vulnerability to the negative impacts of climate change.
The Program is generally being implemented as planned but the lack of ongoing performance data created difficulties in demonstrating the full performance story.
The evaluation found that the comprehensiveness and appropriateness of the Program's performance data could be improved. For example, a performance measurement plan for the Program was developed but this framework is not fully populated and the performance indicators that were identified lack sufficient specificity to provide useful measures of the Program's progress towards achieving intended outcomes. Performance data were available but some of this information was highly scientific and technical such that its complexity limited its usefulness in the evaluation. This evaluation demonstrates that, despite some delays, the Program is being implemented as planned but the lack of ongoing performance information for all program activities and outputs makes it difficult to assess the achievement of expected outcomes.
The following recommendations were developed based on evaluation findings and conclusions. Given that the evaluation demonstrated the relevance of the Program and that program activities are on track for achieving expected outcomes, these recommendations are provided to improve on performance should decisions be made to continue the Program. These recommendations are directed to the Assistant Deputy Minister (ADM) of the Science and Technology Branch.
Develop and implement a plan forward that considers the best utilization of the Program's capacity when establishing priority commitments.
It is not clear that the Program will be able to deliver on all of its expected commitments, particularly for the hazards and extremes component. Available financial resources created challenges for the Program to staff as planned in order to carry out all of its planned research and research-related activities. Therefore, the Program should reconsider its expectations to deliver on all of its commitments. Prioritizing expected deliverables according to departmental needs and the needs of its stakeholders and partners might prove beneficial. Considering existing resource levels, the Program should then explore and implement ways to meet those priority commitments that would have the greatest negative impact if not delivered.
The Program should consider ways to increase their reach to target audiences and enhance their ability to meet target users' climate change information needs, within the context of the Program's capacity.
CCCma, CCCSN and the hazards and extremes component each have a specific target population for their climate change data and information, each with a specialized area of expertise and research interests. The general consensus among stakeholders is that the Program produces good research and its scientists are well-respected in Canada and internationally, but there was concern expressed that the Program's climate change data and information were not reaching as large an audience as possible. The Program, therefore, should explore options to increase reach, such as expanding its partnerships and networks with regional organizations and other federal departments to facilitate increased use and application of its products for CCCma, CCCSN and the hazards and extremes component.
The Program's target audiences have specific climate change information needs and are more likely to use data if they are useful for their needs. The general consensus among Program partners is that more climate change data and information specific to the regions are required, particularly for the North. As well, updated climatic design values are imperative for developing infrastructure that takes extreme climate conditions into account. The Program, therefore, should explore viable options of how to expand its research and related activities to include more climate change data and information that meet these needs. The Program may also consider consulting with stakeholders and partners to identify other key data and information required by target users in order to increase their usefulness further.
Develop a performance measurement framework that includes a clear strategy for ongoing data collection for reporting.
While program objectives were clearly identified in program documents, a clear strategy for measuring progress towards achieving these objectives was not as evident. A performance measurement plan was available but it was not fully populated and performance indicators were not sufficient to provide accurate, ongoing measures of the Program's progress towards achieving outputs and intended outcomes. Therefore, the Program should ensure that a complete performance measurement strategy is developed and implemented that includes specific, measurable performance indicators with specific targets and timelines for measuring progress. When developing these indicators, the Program should explore ways to translate its complex, scientific data into useful information for reporting so that the full meaning of the output being measured is not lost. The Program should also develop a formal mechanism (e.g., a management information system) that organizes these performance measures together, including ongoing financial information, and data collection should be coordinated to ensure that ongoing performance measurement facilitates departmental reporting and future evaluation of the Program's performance.
The ADM of the Science and Technology Branch accepts the evaluation and all of its recommendations. Provided the Improved Climate Change Scenarios Program and associated funding is renewed, the following management actions have been developed in response to these recommendations to improve on activities related to climate modeling, climate change scenarios and hazards and extremes research.
Develop and implement a plan forward that considers the best utilization of the Program's capacity when establishing priority commitments.
Management agrees with the recommendation.
The evaluation showed that the Program is, by and large, on track to being implemented as planned; however, in order to allow completion of work on hazards and extremes that was delayed at the outset, the Program plans to meet all of its remaining priorities by seeking program renewal beyond March 31, 2011 given that the evaluation has confirmed the relevance of the program. A plan for priority commitments will be established, wherein commitments will focus on the continuation of climate model development and further increases to the resolution of regional climate model projections to meet target audiences' information needs. Priorities will be provision of climate data and information to specific regions, including the North, bridging and translation of new climate science on scenarios, hazards and extremes into relevant and accessible data products and tools for decision-makers, and continuation of partnerships to complete program deliverables on hazards for disaster management and infrastructure codes and standards. As part of this process, the Program will also consult with other sections within Environment Canada on cross-cutting activities that could be included in the plan forward under renewed funding. In addition, this consultation will address the broader issue of how best the federal government can facilitate the uptake and application of climate and adaptation science by target populations.
Should the Program not be renewed beyond March 31, 2011 the Program will continue to focus on its current priorities, taking into account the Program's current capacity, from now until March 31, 2011.
Timeline
March 31, 2011
Deliverable(s)
Assessment of Program's current capacity and delivery of priority commitments.
Responsible Party
Director General, ASTD
Within one year of commencement of new funding (pending future funding decisions)
A plan for addressing priority commitments under renewed funding.
Evidence of consultations within Environment Canada (e.g., meeting minutes, memos, briefing notes).
Director General, ASTD
The Program should consider ways to increase their reach to target audiences and enhance their ability to meet target users' climate change information needs, within the context of the Program's capacity.
Management agrees with the recommendation as a suitable expectation for a renewed Adaptation program.
The Program will continue to monitor stakeholder and partner information needs, and consider ways to increase its reach to target audiences and enhance its ability to meet target users' climate information needs. These may include investing more heavily in delivery mechanisms, such as improved and more comprehensive provision of climate change information and supporting scientific data via web sites that have improved user interfaces, and by strategic implementation of user training workshops to reach target populations ready and capable of assimilating the information.
It is important to note, however, that, to date, targeted stakeholder consultations with codes and standards agencies and provincial disaster management agencies have been completed. Training sessions on the use of climate change scenarios also continue to take place across Canada, including sessions provided upon request to stakeholders. In addition, over the coming months the new regional climate model will be in a position to provide more detailed region-specific information for all Canadians.
Working within the broader EC community, will ensure that programs are developed to meet information and training needs of target populations, taking into consideration both resource and staffing issues. It would not be realistic to expand Program activities to increase its reach to target audiences and enhance its ability to meet target users' climate change information needs without renewed funding.
Timeline
March 31, 2011
Deliverable(s)
Consultations with stakeholders and partners about information needs.
Responsible Party
Director, CRD
Director General, ASTD
Within one year of commencement of new funding (pending future funding decisions)
Co-lead the development of an intra-interdepartmental strategic plan for improved client service through participation in meetings and decision-making processes.
Improved delivery and accessibility of outputs to enhance user up-take of information in target populations.
Director, Climate Research Division, ASTD
Develop a performance measurement framework that includes a clear strategy for ongoing data collection for reporting.
Management agrees with the recommendation as a suitable expectation for a renewed Adaptation program.
On April 1st, 2010 the Department instituted a new Performance Measurement Framework (PMF) which includes detailed performance measurement reporting for both outcomes and outputs. The 2010/11 PMF outlines targets at the Output, Expected Result, and Strategic Outcome levels.
At the Strategic Outcome, Expected Result and Output levels performance indicators have been identified for the hazards and extremes components of the Program e.g.: science input to new infrastructure/building codes, and climate projections and climate data sets on various time and spatial scales. These indicators are in line with the recommendations from the 2006 report by the Office of the Auditor General (CESD). At the Output level, a performance indicator for the climate modelling and projections component has also been identified. Accordingly, the data collection needed to report on these PMF indicators is now integrated within this overarching Departmental structure.
Should the Program be renewed, it will seek additional opportunities and mechanisms to meet the near and longer term information needs of target populations, particularly in the context of climate service delivery, to which the Program would be an important contributor.
The Program will carefully consider how to measure the performance of climate scientific research programs in a manner that takes into consideration the fact that the impacts of this research often take many years to manifest, and are dependent upon the capacity and abilities of target populations to assimilate. Therefore performance measurement strategies must focus on longer term performance measures dependent on capacity of target populations to change behaviour, and not solely individual programs.
Timeline
Completed
Deliverable(s)
PMF for 2010/11.
Responsible Party
WES Board Secretariat
March 31, 2011
Fully populated PMF for 2010/11 for the Output, Expected Result
Evidence of continued data collection and reporting under the recently adopted PMF (e.g., web use statistics, surveys, stakeholder evaluations).
Director, CRD
Within one year of commencement of new funding (pending future funding decisions)
The Program will develop a performance measurement framework encompassing longer time frames, in the context of Canadian adaptive capacity.
Director General, ASTD