Home > Proactive Disclosure > Audits and Evaluations > A&E Reports > 2010-2011 > Evaluation of the Improved Climate Change Scenarios Program

Evaluation of the Improved Climate Change Scenarios Program

Final Report
November 2010

Previous Page | Table of Contents | Next Page

4.0 Findings

This section presents the findings for each evaluation issue (i.e., relevance and performance) and its related evaluation questions. (Refer to Annex 2 for the Evaluation Matrix.) The findings at the overall issue level are presented first, followed by the findings for each evaluation question.

A rating is also provided for each evaluation question based on evaluation findings. The rating symbols and their significance are outlined below in Table 2. A summary of ratings for the evaluation issues and questions is presented in Annex 4.

Top of Page

4.1 Relevance

Evaluation Issue: Relevance

Top of Page

4.1.1 Issue 1: Continued Need for the Program

1. Are activities within the Program connected with key environmental climate change needs?

• Indicator(s): Degree to which the Improved Climate Change Scenarios Program continues to meet climate change adaptation needs
• Methods:
  • Document review
  • Key informant interviews
• Rating: Achieved

Research activities within the Improved Climate Change Scenarios Program are connected with key environmental climate change needs.

• The impacts of climate change are already evident through global warming,^[18] increases in the number, duration and intensity of heat waves and heavy precipitation events,^[19] changing precipitation patterns and river flows, and melting Arctic sea ice and permafrost,^[20] all of which will have significant implications for Canada's communities and infrastructure. For example, longer and more intense heat waves and smog events increase the potential for adverse effects on health, while coastal erosion from rising sea levels, more severe winter storms and more frequent flash floods pose greater risks to safety and property.^{[21], [22]}
• The literature identifies the need for Canada to develop a co-ordinated process to prioritize climate change risks and plan for adaptation in response to these climate change impacts,^[23] noting the need for the types of research developed through the Improved Climate Change Scenarios Program such as analyses of the impacts of climate change as well as climate science activities like climate modelling. ^[24]
• Program documentation demonstrates consistency between program activities and meeting these climate change information needs. For example, peer-reviewed publications and working papers by program scientists address information gaps and identify areas of potential risk to infrastructure and public safety. As well, climate science information produced by the Program through its climate modelling activities and research on adaptation and impacts are made available through the development and maintenance of the CCCma, CCCSN and hazards websites.
• Key informants report that the Program's research activities contribute foundational data and information for decision making. In their view, this helps to reduce the increased vulnerability of Canadians and Canadian communities over the long term due to climate change and extreme weather, particularly in terms of existing infrastructure.
• Interviewees report that the Program contributes to meeting climate change needs by providing access to data and information via program websites and training sessions.
  • External partners identified a dependence on climate modelling and adaptation and impacts research in order to perform their own work (e.g., academics, other researchers and scientists, engineers, municipal planners). As well, interviewees report that having a Canadian climate model run by Canadian scientists helps to develop scientific capacity domestically and also gives Canada a leading role within the international scientific community; and
  • Partnerships and networks established through the Program's activities are viewed as contributing greatly to the larger climate science community, particularly in the ways that these links feed other organizations like Ouranos, the National Research Council (NRC) and the Canadian Standards Association (CSA).
• Evidence from interviews, however, also identifies some program shortfalls in terms of meeting climate change needs. For example, program products (e.g., website tools) are viewed as being underutilized, although the reason for this is unclear; climate modelling and data on Canada's North are inadequate; and information that feeds into infrastructure design needs to be updated to account for climate change.

Top of Page

4.1.2 Issue 2: Alignment with Government Priorities

2. Are activities within the Program aligned with federal government priorities?

• Indicator(s): Degree of alignment between the objectives of the Improved Climate Change Scenarios Program and federal government priorities related to adaptation to climate change
• Methods:
  • Document review
  • Key informant interviews
• Rating: Achieved

The objectives of the Improved Climate Change Scenarios Program are aligned with larger federal government priorities through their contributions to climate change science and to adaptation and impacts research.

• The Government of Canada priorities outlined in the 2007 and 2008 Speeches from the Throne and 2007 Budget identify the federal government's commitment to addressing climate change issues. While the 2009 Speech from the Throne does not refer specifically to adaptation or climate change, it does commit to building Canada through new investments in infrastructure as part of its economic stimulus plan.
  • Evidence from both program documentation and interviews indicates that the Program has actively participated in consultations with national organizations that form and publish the National Building Code of Canada (NBCC) to update codes and standards to include more recent climatic information for extreme weather resilience for new and retrofitted infrastructure.
• As well, Canada is a signatory to the UN Framework Convention on Climate Change (UNFCCC) which legally commits all parties to formulate, implement, publish and update adaptation measures. According to interviewees, the Program provides the climate science as well as adaptation and impacts research from which these adaptation measures may develop. As well, the Program supports federal international commitments through its research contributions to the IPCC Fourth Assessment Report. The Program was working to contribute scientific climate research to the IPCC Fifth Assessment Report at the time of data collection.

3. Are activities within the Program aligned with the priorities of Environment Canada?

• Indicator(s):
  • Extent to which the objectives of the Improved Climate Change Scenarios Program match mandate and priorities of Environment Canada
  • Perceptions regarding the degree of alignment and examples of alignment between priorities of Environment Canada and the Improved Climate Change Scenarios Program
• Methods:
  • Document review
  • Key informant interviews
• Rating: Achieved

There is evidence that the objectives of the Improved Climate Change Scenarios Program are strongly aligned with Environment Canada priorities.

• Evidence from program documents demonstrate that program activities, outputs and intended outcomes are aligned with Environment Canada's Departmental Strategic Outcome that Weather and environmental predictions and services reduce risks and contribute to the well-being of Canadians. More specifically,
  • CCCma's program activities in the development and enhancement of global and regional climate models provide scientific climate information to contribute to the departmental priority to improve knowledge and information on weather and environmental conditions to influence decision making; and
  • AIRS' research activities on adaptation and climate change impacts address the departmental priority to ensure that Canadians are informed of, and respond appropriately to, current and predicted environmental conditions. AIRS' adaptation science plan, which describes its mission, strategic directions and implementation and measurement strategies, clearly describes the alignment between its activities and this departmental priority and how AIRS contributes to the larger scientific research agenda for the Department.^[25]
• The majority of key informants felt that the Program is aligned to a great extent with departmental priorities but some felt adaptation should be an even greater priority for the Department.

Top of Page

4.1.3 Issue 3: Alignment with Federal Roles and Responsibilities

4. Is there a legitimate and necessary role and responsibility for the federal government in the Improved Climate Change Scenarios Program?

• Indicator(s):
  • Absence/presence of similar programs within the federal and/or provincial governments
  • Extent to which similar programs complement or duplicate the Improved Climate Change Scenarios Program
  • Perceptions and examples of gaps in meeting the climate change needs of Canadians without the Program
• Methods:
  • Document review
  • Key informant interviews
• Rating: Achieved

Evaluation evidence demonstrates that there is a legitimate and necessary role and responsibility for the federal government to deliver the Program as it provides a unique service, is well positioned to overcome jurisdictional boundaries so that necessary information can be provided at a national and regional level, and is a trusted source of the information. The Improved Climate Change Scenarios Program complements other similar programs, providing information and services that are unique within the climate change science network and contribute to the success of complementary programs.

• Interviewees felt that the Government of Canada has a responsibility to ensure the safety and security of Canadians from threats of environmental harm (including climate change impacts on infrastructure) as well as negative impacts on the economy. The federal government is believed to be well suited for this role because it is able to provide the large amount of scientific expertise, resources, infrastructure and capacity (including technical capacity) needed to address climate change impacts in the long term. As well, there is a need for centralization to provide consistent information and quality data from an independent, national perspective that can contribute to the international scientific community.
• Evidence from documents supports this view. The 2006 Report of the Commissioner of the Environment and Sustainable Development (CESD) contends that the federal government has a necessary role and responsibility in carrying out climate modelling, establishing and operating climate monitoring networks, conducting impacts and adaptation research in response to policy needs and providing national leadership in the direction, co-ordination, and funding of climate science in Canada.^[26]
• In addition, NRTEE recommends that the federal government dedicate resources to reliably update and disseminate regionally relevant climate data and information, climate change projections and climatic design values^[27] to support infrastructure decisions. NRTEE further notes that there are many implications for Canada's North related to monitoring capacity and that there has been a reduction in federal involvement in developing and updating climatic design values. The report also notes the reluctance of organizations that develop and update Codes, Standards and Risk Instruments (CSRIs) to use climatic design values derived by any group except Environment Canada, which they perceive as an expert, reliable, neutral third party.^[28]
• Interviewees generally viewed other programs and organizations involved in the same or similar activities (for example, the Pacific Climate Impacts Consortium (PCIC) in B.C. and the North American Regional Climate Change Assessment Program (NARCCAP) in the U.S.) as being complementary rather than duplicating the Improved Climate Change Scenarios Program. Reasons provided included observed differences in activities between the Program and these other programs or groups; the national focus of the Program compared to the regional focus of other research organizations; and the need for research to build on or duplicate previous analyses to ensure the robustness of results (e.g., most scientists will encourage the use of multiple models in climate science research because of the uncertainty inherent in each model's design). One interviewee, however, suggested that there could be a better division of labour between the federal government and regional organizations with regard to the work being done.
• Interviewees also felt that, in the absence of the Program, there would be gaps in available information both in Canada and internationally, lack of consistent or national coverage that focuses on broad issues, and an absence of national standards for infrastructure which would result in a high degree of liability for organizations. Interviewees felt that these gaps would lead to loss of awareness, knowledge and expertise in the scientific community as well as an increase in property damage, lost productivity and negative impacts on the safety and security of Canadians.
• Interviewees generally felt that there is no one ready or able at this point to fill any gaps that would arise should the Program cease to function, contending that Environment Canada is the only organization with the capacity, expertise and mandate to address adaptation needs on behalf of Canada and Canadians. The reasons given focused on areas such as the large amount of investment, resources and/or capacity required; the time required to put things in place (e.g., software infrastructure); and the regional focus of some other organizations involved in this area. The challenge in obtaining Canada-specific information from international sources was also mentioned.

Top of Page

4.2 Performance

Evaluation Issue: Performance

Top of Page

4.2.1 Achievement of Expected Outputs and Outcomes

The following sections (4.2.1.1 to 4.2.1.4) describe the evidence and ratings for the achievement of expected outputs and outcomes for the Improved Climate Change Scenarios Program.

4.2.1.1 Achievement of Expected Outputs (Issue 4: Achievement of Expected Outcomes)

5a) To what extent have intended outputs been achieved within the Program?

• 4.2.1.1.1 Tools, Processes and Systems

  • Indicator(s):
    • Evidence of enhanced global climate model and development of regional climate models
    • Evidence of development of tools for climate change scenarios
    • Evidence of updates to scenarios and hazards websites
  • Methods:
    • Document review
    • Key informant interviews
  • Rating: Achieved

  Evidence from document review and interviews indicate that the Program is producing its intended tools, processes and systems for each of the components of the Improved Climate Change Scenarios Program, although some deliverables for the hazards and extremes component were not evident at the time of data collection.

  • For program outputs in climate model development, evidence demonstrates that extensive work has been completed on the development of and improvements to, climate models. For example, CCCma is carrying out tests for Canada's new Earth System Model (Can ESM2), is making improvements to other climate models such as a Canadian Terrestrial Ecosystem Model (CTEM) and the Canadian Model of Ocean Carbon (CMOC), and has conducted numerous experiments on regional climate models in collaboration with university partners and consortiums.^[29] While internal partners and stakeholders agree that the Program is achieving its outputs for global and regional climate models, other government departments feel that climate information for the North is lacking and requires special attention.
  • Evaluation findings also demonstrate evidence of the development of tools for climate change scenarios. Key informants noted that the Program is providing ongoing data updates and identified the development of software tools for mapping and an online tool for developing bio-climate profiles on the CCCSN website that facilitate access to climate information. The 2009‑2010 mid-year review for AIRS notes that the Program is on track for making intended improvements to the network's website and for developing new scenario validation methods with Canadian academics and international partners.
  • Documents also reveal that updates and improvements have been made to the Hazards website that have included additional images, maps and data on snow, floods, heat and cold. Briefing documents for senior management suggest the Program is on track but most program staff and some internal partners felt that the evaluation was taking place too early to be able to state conclusively that the hazards and extremes component was achieving its intended outputs. Nevertheless, they did agree that it was progressing adequately.
 

• 4.2.1.1.2 Research

  • Indicator(s):
    • Number of peer-reviewed publications
    • Program outputs in international assessments
    • Evidence of the development of new research based on new climate models
    • Evidence of methodologies examined for incorporating changing climate into infrastructure design
  • Methods:
    • Document review
    • Key informant interviews
  • Rating: Achieved

  Evidence clearly shows that research is being produced by each of the areas within the Improved Climate Change Scenarios Program.

  • A total of 157 peer-reviewed publications were published by CCCma and AIRS^[30] between 2007 and 2009. The number of publications by program area is presented in Table 3.

    Table 3: Number of publications by Section

    	Number of Peer-Reviewed PublicationsA
    A Peer-reviewed publications include journal articles, books, book chapters, reports and occasional papers. B Data on publications were provided for AIRS as a whole and could not be separated by CCCSN and Hazards research.
    CCCma	45
    AIRS B	112
    Total	157

  • As well as producing research for publication, scientists from both CCCma and AIRS contribute to the international scientific community through their ongoing participation in IPCC assessment reports. Both internal and external partners indicate that published research on climate models by CCCma scientists contributed to the generalized conclusions reported in the IPCC assessments and view this as a strength of the Program. Documentation supports this finding, demonstrating that the CCCma global climate model was used to generate climate change scenarios for the 2007 IPCC Fourth Assessment Report.
  • In addition to contributing to the international scientific community through involvement in the IPCC, the Program conducts research to expand knowledge and understanding of climate change. Interviews with program staff indicate that experiments have been conducted on the Canadian global climate model and that papers based on this research have been submitted or published. This is supported through documents that provide evidence of experiments and model runs for further enhancements of Canada's global climate model. Furthermore, documents show that CCCma's Canadian global climate models are among the top ranking models within the field.^{[31], [32]}
  • Program staff and external partners report that the development of enhanced methodologies for incorporating changing climate into infrastructure design is in progress. Program documents also demonstrate that research and development of proposed codes and standards are being carried out for the 2010 National Building Code. A 2009‑2010 mid-year review for AIRS, however, notes that delays in staffing led to delays in the development of new models and methodologies for analyses of extreme wind and flood events. Delays in research related to settlements in the North were also experienced, although no reasons for this delay were provided. In contrast, the development of new analytical approaches for climate change, infrastructure and transportation in southern Canada is reported to be on track.
 

• 4.2.1.1.3 Partnerships and Networks

  • Indicator(s):
    • Evidence of participation in research networks or committees
    • Number of CCCSN servers deployed by region and by location
    • Number of CCCSN training workshops conducted
    • Number of regional sites added to hazards website
    • Evidence of stakeholder organizations consulted
  • Methods:
    • Document review
    • Key informant interviews
  • Rating: Achieved

  There is strong evidence that the Program has engaged partners and established or expanded their networks, although some challenges were identified.

  • Evidence from both interviews and program documents show that the Program participates in formal research networks (e.g., Global Ocean-Atmosphere Prediction and Predictability [GOAPP] Network), engages with partners on research projects (e.g., with academics, for Canadian Stratospheric Processes and their Role in Climate [C-SPARC] and Canadian Network for Regional Climate Modelling and Diagnostics [CRCMD]) and serves on advisory or steering committees or task forces (e.g., for Pacific Climate Impacts Consortium [PCIC], Climate Change Task Force for the Transportation Association of Canada, National Building Code of Canada's [NBCC] Climatic Loads Task Group).
  • For research on climate change scenarios, the development of research partnerships is facilitated through the CCCSN. CCCSN nodes are groups of "working teams" based at different physical locations with super-computer capacity to conduct climate change research. Plans were to establish six regional nodes (Pacific, Yukon, Prairie, Ontario, Quebec and Atlantic nodes), each with hardware (computer servers) and access to data as well as funding to develop regional expertise and research. Interviews with program staff and briefing documents to senior management indicate that the Program is on track to increasing the number of regional nodes as planned. Program staff reports that servers have been deployed to all regional nodes except for the Pacific and Atlantic nodes. Regional research was in progress but websites were not yet functional, although the Program reported being on schedule for finalizing the websites and providing public access to them.
  • CCCSN also conducted training sessions where participants learn about what is available on the CCCSN website and how climate change information can be displayed, downloaded, validated and interpreted using tools available on the CCCSN website. Evidence shows that the CCCSN conducted eight regional training sessions that were carried out in Toronto, Regina, Fredericton and Whitehorse within the scope of this evaluation. Only two sessions had been planned, but additional sessions were added (using AIRS O&M funds) at the request of CCCSN website users. Key informants felt that these sessions were a good opportunity for scientists with common information goals to network with one another.
  • For the Hazards network, the Ontario, Atlantic and Prairie and Northern nodes of the Canadian Atmospheric Hazards network were running at the time of data collection, while the Pacific and Yukon as well as Quebec nodes were still waiting release. These three nodes were established during the analysis and reporting phase of the evaluation so that the national Hazards network is now complete.
  • Data from both interviews and documents also provide evidence that  partnerships and networks for the work on hazards and extremes have been expanded through consultations that were held with stakeholder organizations. For example, consultations were completed with the Conservation Authorities (CAs) of Ontario, the Transportation Association of Canada, the Arctic Council, and the Canadian Telecommunications Preparedness Emergency Association.
  • Formalized agreements are another mechanism to facilitate collaboration. At the time of data collection, one formal Memorandum of Understanding (MOU) had been established between the Program and with CSA to work together on developing standards and codes but interview data suggest that other formalized partnerships are in development.
  • Some key informants, however, believe that the Program's ability to increase and expand their partnerships and networks may be limited.
    • Program staff feel that they need more FTEs with relevant subject matter expertise to manage data and information requests from partners and stakeholders. The demand is extensive but the supply of scientists with relevant expertise who can provide this service is limited. The Canadian Meteorological and Oceanographic Society (CMAO) estimates that, taking into account all government, private sector and academic employment, there are approximately 1200 people across Canada with some atmospheric science background.^[33] This creates additional pressures when staffing because the pool of qualified candidates is small. The Program has expanded its search internationally, with some success, to draw expertise back into Canada.^[34]
    • According to external interviewees, academics that were once part of the CCCma research community are no longer able to participate due to funding cuts from federal organizations such as National Sciences and Engineering Research Council (NSERC). This factor is external to the Program but points to the vulnerability of the Program in terms of accessing expertise.
    • One regional partner felt that, although the connections that do exist are likely good, there are some gaps in the kinds of partnerships and networks that could be developed through the CCCSN. For example, CCCSN could improve on their partnerships with other federal departments (such as NRCan) so as to improve communication and coordination of research activities.

Top of Page

4.2.1.2 Achievement of Expected Immediate Outcomes (Issue 4: Achievement of Expected Outcomes)

5b) To what extent have immediate outcomes been achieved within the Program?

• 4.2.1.2.1 Increased awareness of, availability of and access to research, websites and climate models

  • Indicator(s):
    • Number of data downloads and data requests from program websites
    • Evidence of increased awareness of, availability of and access to global and regional climate models
    • Evidence of increased awareness and improved knowledge on climate models among training sessions and workshop participants
    • Evidence of improved access to hazards websites
  • Methods:
    • Document review
    • Key informant interviews
    • File review
  • Rating: Progress made, attention needed

  Data suggest that there has been some progress in increased awareness of, availability of and access to models, websites and research among the users of the Program's products. However, continuous work is needed to increase the reach of users.

  • Program staff feel that there is increasing awareness of, availability of and access to program outputs, as demonstrated by the increasing number of data downloads and data requests for each of the Program's websites. The Program collects ongoing performance data on website usage for the CCCma, CCCSN and Hazards websites. A review of program files on website usage indicates that the number of data downloads for CCCma and the number of data requests for CCCSN and the Hazards websites has increased since 2007, with the exception of CCCma, which showed a 20% decline in data downloads between 2007 and 2008.^[35] (Please refer to Table 4 for distribution across years for each program component.) These observed increases suggest that target users were aware of program websites and the available information, and were able to access the data.

    Table 4: Number of Data Downloads and Data Requests by Year

    	2007	2008	2009	Total
    A Number of data downloads by users. B Number of data requests by users.
    CCCma website A	14,854	11,858	15,952	38,785
    CCCSN website B	787,296	1,372,335	2,185,445	4,345,076
    Hazards website B	264,135	866,669	1,041,691	2,172,495
    Total	1,066,285	2,250,862	3,243,088	6,560,235

  • Program staff feel that they have increased their target audience's awareness of and access to Canada's global climate model and the regional climate models. They report that they have achieved this through their research, the results of which are made available through publications, presentations and posting on the CCCma website for viewing and downloading. Internal and external partners who work with CCCma also agree with this view, noting that CCCma has helped improve their knowledge of models by providing expert advice, publishing peer-reviewed research and presenting at conferences and workshops. Some interviewees note, however, that the highly technical and specialized nature of this climate model information creates challenges in making this information useful for decision-making.
  • This increased awareness is also evident through the training sessions that were conducted by CCCSN officers. These hands-on, regional training sessions were conducted across Canada to train participants on the use of CCCSN products, such as climate change scenarios data, as well as to promote availability and access to the hazards and climate extremes information on the Hazards website.^[36] Participant evaluations of a series of training workshops held^[37] during the course of this evaluation show that strong majorities of participants rated the overall workshops (89%) and their content (83.5%) as "very good" or "excellent", and felt the AIRS instructors were very helpful, approachable and knowledgeable, with a great deal of subject matter expertise. Some interviewees, however, expressed concern that CCCSN was restricted by the number of program staff available to conduct these sessions and answer questions posted by users on the CCCSN website, ^[38] which may be a limiting factor in the Program's performance.
  • This limited capacity was also identified for the hazards and extremes component of the Program. Although interviewees generally felt that the Program has contributed to increased awareness of, availability of and access to climate extremes research and hazards information through the Program's ongoing information sessions with key stakeholders and partners, external partners suggest that technical committees for developing infrastructure do not have updated data and information on climatic design values because the data that feed into these values are not available. This lack of data is corroborated by evidence from the Program's progress report to senior management that indicates extreme event analyses to support the development of new national codes and standards have been delayed. These delays are attributed to staffing issues. At the time of interview, only 2 of the 12 FTEs positions were staffed to carry out activities on hazards and extremes. This lack of data could have a serious impact on the ability of target users to make informed decisions on infrastructure designs, which could ultimately have a negative impact on the safety of Canadians.
 

• 4.2.1.2.2 Greater collaboration within Canada and internationally to address climate change issues

  • Indicator(s):
    • Evidence of collaboration with key stakeholders and program partners, as demonstrated by:
      • Co-authored publications
      • Formal agreements
      • Engagement with standards and codes agencies and industry associations
  • Methods:
    • Document review
    • Key informant interviews
  • Rating: Achieved

  Overall, the Program's established partnerships and networks led to greater collaboration to address climate change issues.

  • This evaluation examined how the Program's partnerships and networks contributed to observed increases in collaboration with key stakeholders and partners to address climate change issues. As observed by some interviewees, collaboration to address climate change issues may not be solely attributed to the Program because there is a long history of climate modelling and climate science in Canada and research organizations in this field have existed for a long time. Professional collaboration may have been well established prior to the beginning of the Program and some of this collaboration may result from personal networks and informal contacts within the scientific community rather than directly linked to program outputs.
  • Overall, increased collaboration as a result of the Program's established partnerships and networks was demonstrated in the following ways:
    • The Program's involvement in international organizations such as IPCC and the North American Regional Climate Change Assessment Program (NARCCAP) facilitated collaborative research with partners in the international scientific community. Program staff and internal partners feel that this involvement in international committees and projects has resulted in increased awareness, recognition, influence and respect for Canadian climate modelling and research on adaptation and climate change impacts within the international scientific community.
    • Engagement with agencies, groups and consortiums working on climate impact and adaptation issues has also facilitated collaboration. Interviews with program staff and external partners demonstrate that the Program's partnerships with industry and professional associations have resulted in collaboration among partners and contributed to building safe infrastructure.
  • Interview data also indicate that collaboration among key stakeholders and programs was initiated or expanded further as a result of the Improved Climate Change Scenarios Program activities and outputs. External partners note that availability and access to the research and tools produced by the Program as well as knowledge and expertise provided by program officers contributed to the higher level of collaboration they experienced with their own stakeholders in their own work.

Top of Page

4.2.1.3 Achievement of Expected Intermediate Outcomes (Issue 4: Achievement of Expected Outcomes)

5c) To what extent have intermediate outcomes been achieved within the Program?

• 4.2.1.3.1 Increased use of websites and climate models among target populations

  • Indicator(s): Perceptions among target populations on use of climate models and program websites in their own work
  • Methods:
    • Document review
    • Key informant interviews
  • Rating: Progress made, attention needed

  It is not clear from the evidence how data and information accessed from Program websites are being used. Potential users may not have the capacity to use this information and the usefulness of the information available may be improved with more focus on regional or local climate data.

  • Program records indicate that the Program's websites are being increasingly accessed over the last two years. (Refer to Table 4 for frequency of data access for each program component.) However, while access to data and information on Program websites increased, it is difficult to determine how it is being used. External partners report that, with the exception of climate science experts, data and information available on these sites may be difficult for some target populations (e.g., industry and other levels of government) to use without assistance from program staff. Similarly, interviews suggest that not all potential users of the Program's products are equipped to use them and require this information to be available in a useable form. As a consequence, while scientific researchers reportedly access and make use of the data, other target groups with less technical or scientific capacity (e.g., municipalities) seek out information from other sources.
  • The level of use of climate change data and information may also vary according to their perceived usefulness. Evidence demonstrates that external partners are more likely to find regionally or locally focussed information more useful to their work than national or global information. For example, respondents to a survey on community access and use of climate information^{[39], [40]} were much more likely to rate regional or local climate projections than national or global climate projections as "very useful" (77% versus 23%, respectively). This finding is consistent with evidence from interviews with external partners, who reported that there was a greater use of information from climate change networks with a regional or local focus (e.g., PCIC, the Toronto Urban Climate Change Network).
 

• 4.2.1.3.2 Increased capacity to conduct climate change impacts and adaptation research among target populations

  • Indicator(s): Demonstrated evidence of increased capacity to conduct climate change impacts and adaptation research as a result of the Program
  • Methods:
    • Key informant interviews
    • Document review
  • Rating: Progress made, attention needed

  Evidence indicates that progress is being made towards increased capacity for target groups to conduct climate change impacts and adaptation research, but more needs to be done.

  • As for increased use of websites and climate models, increased capacity for target populations to conduct climate change impacts and adaptation research largely depends on the target group. Interviewees report that availability of downscaled models and training sessions to improve access to the websites that were provided by the Program has contributed to increased capacity among scientists to conduct their own climate change impacts and adaptation research. Documentation demonstrates that community leaders still require the expertise of program officers to use and apply specialized climate information as well as to understand this information in a broader context when developing strategies to reduce vulnerability to climate change. However, the degree to which the department is responsible for the capacity building of target users is not clear.
  • External factors that affect the capacity of target populations to conduct their own research revolve around available resources. External partners noted that the lack of officers with sufficient expertise and the limited financial resources within their own organizations impedes their capacity to utilize climate information and to conduct adaptation and impacts research. As well, the instability of federal funding from other sources such as the Canadian Foundation for Climate and Atmospheric Science (CFCAS) negatively affects the capacity of climate modelling scientists to conduct their research.
 

• 4.2.1.3.3 Increased application of adaptation science and climate change research among target populations and appropriate jurisdictions

  • Indicator(s): Demonstrated evidence that adaptation and climate change impacts research has been applied
  • Methods:
    • Key informant interviews
    • Document review
  • Rating: Progress made, attention needed

  Although there is some evidence demonstrating that target groups have applied adaptation and climate change research in their own work, it is too early to determine whether or not this outcome has been fully achieved.

  • Almost half of key informants report that it is too early to assess whether or not this outcome has been achieved, suggesting that research may be applied more fully over the next 5 to 10 years to improve Canada's infrastructure design to account for climate change impacts. Documents, however, suggest that this application of research has begun. A 2009 survey finds that 61% of communities report the Program's climate information is useable for their purposes but there are still only a few communities making use of this information. For example, cities like Ottawa, Toronto and Vancouver have used adaptation and impacts research for planning purposes at the local level (e.g., for work on freezing rain and sea walls).
  • While the Program has worked towards making information available and teaching target groups how to access and use the data, interviewees feel there is still limited application of this research in Canada. Overall, interviewees note that there needs to be more work on raising awareness of the impacts of climate change and educating Canadians on the need to adapt before adaptation science and climate change research can be applied. Increased awareness of and education on scenarios information and climate science research are a foundational contribution that will lead to application.

Top of Page

4.2.1.4 Achievement of Long-Term Outcomes (Issue 4: Achievement of Expected Outcomes)

5d) To what extent have long-term outcomes been achieved within the Program?

• 4.2.1.4.1 User groups apply tools and information to assess climate change risks and plan adaptation strategies

  • Indicator(s):
    • Opinions among user groups that research and related products from the Program were used to assess risk and inform the development of strategies and plans to prepare for future climate extremes
    • Evidence of the adoption of new building codes that consider climate change risks
  • Methods:
    • Document review
    • Key informant interviews
  • Rating: Too early to observe achievement

  Evidence suggests that the Program is generally on track to achieve this long-term outcome but it is too early to expect to observe the full achievement of this outcome.

  • Progress is being made in the application of research and climate information among user groups to assess climate change risks and inform the development of adaptation strategies to plan for future climate extremes. For example, 84% of communities in a 2009 survey reported that they were developing strategic plans to adapt to climate change^[41] and the Toronto Region Conservation Authority (TRCA) is working with climate information to better plan and prepare for future climate variability.
  • Interviewees generally felt that progress is being made among user groups in adopting new building codes that consider climate change risks. For example, interviewees report taking intense rainfalls into account when redesigning the drainage of roads in order to adapt infrastructure to the impacts of climate change, and the Canadian Standards Association (CSA) has used CCCSN data, among other data sources, to draft guidelines on infrastructure foundations for permafrost in Canada's North. As well, the Toronto Region Conservation Authority (TRCA) indicates that it has been working with Environment Canada, several municipalities and multiple conservation groups on a pilot study to integrate climate change impacts into water-related infrastructure design practices.^[42] Interviewees, however, generally felt that more needed to be done to incorporate climate change risks into infrastructure designs.
 

• 4.2.1.4.2 Stakeholders have taken actions to reduce their vulnerabilities from, and have adapted to, predicted impacts of climate change

  • Indicator(s):
    • Perceptions that behaviours have changed as a result of the Program
    • Examples of user groups that factor climate change risks into their planning/decision processes
  • Methods:
    • Document review
    • Key informant interviews
  • Rating: Too early to observe achievement

  Evidence suggests that it is too early to expect to observe the full achievement of this outcome but there is some evidence demonstrating that the Program has contributed to adaptive behaviour.

  • Overall, interviewees felt that it is too early in program implementation to observe adaptive behaviour to reduce vulnerabilities to the impacts of climate change. Some interviewees, however, do identify a link between the Program and observed adaptive behaviour, although some connections are stronger than others. Regional and municipal partners in both government and industry report that the Program's work has contributed to stakeholders taking actions to reduce their vulnerability to climate change impacts (e.g., promoting personal emergency kits to prepare for major blackouts, developing an extreme heat program, making changes to regional operations to address increased risk for wildfires).
 

• 4.2.1.4.3 Risks to communities, infrastructure and to the health and safety of Canadians resulting from climate change have been reduced

  • Indicator(s): Evidence of whether risks to the health of Canadians resulting from climate change have been reduced as a result of the Improved Climate Change Scenarios Program
  • Methods:
    • Document review
    • Key informant interviews
  • Rating: Too early to observe achievement

  Evidence suggests that the Program is generally on track to reduce risks to communities, infrastructure and to the health and safety of Canadians resulting from climate change but it is too early to tell the extent to which this outcome has been achieved.

  • Interviewees, including program staff as well as internal and external partners, felt that some progress has been made and/or the Program is on track to achieve this intended outcome but were unable to report definitively that risks have been reduced. Although a few examples were provided by interviewees, they related more closely to awareness of risks than to risk reduction. In addition, the extent to which these examples could be linked to work done by the Program during the timeframe covered by the evaluation was not clear.
  • Some documents indicate that codes, standards and guides are being developed nationally but documentation was generally quite limited in identifying the risks that have been reduced.

Top of Page

4.2.2 Implementation of the Program (Issue 4: Achievement of Expected Outcomes)

6. Has the Program been implemented, or is it on track to being implemented, as planned? Are program activities, processes and governance structures adequate for achieving expected program results?

• Indicator(s):
  • Extent to which performance data is comprehensive and appropriate
  • Proportion of activities and outputs planned versus those completed
  • Perceived adequacy of activities, processes and governance structures for achieving expected program results among program staff and partners
  • Extent to which barriers to program implementation are identified and/or resolved
• Methods:
  • Document review
  • Key informant interviews
• Rating: Progress made, attention needed

Program documents reveal that the Program is generally being implemented as planned but it is not clear whether or not complete implementation will take place within the Program timeframe, given the incomplete performance data available and given the delays experienced by CCCSN and the hazards and extremes component.

• Overall, the Program's performance measures were neither comprehensive nor appropriate. Of the performance data that were available, some were well-maintained and up to date (e.g., number of data downloads and requests for data and information from program websites, evaluations from CCCSN training sessions) while most were not well-organized or useful. Part of the challenge may be in converting the Program's highly technical and scientific activities and outputs into a form that is easily understandable for reporting and evaluation purposes. Program information was provided when requested but there was no clear indication that the Program was collecting performance information on an ongoing basis. Although a performance measurement plan for the Program is available, its performance indicators are difficult to measure, are not specific enough to provide an adequate assessment of the Program's performance^[43] and do not include target values or baseline information. Some mid-year reviews and periodic briefings to senior management provided information on the Program's progress but these documents were not consistently provided for each fiscal year and for each program component, and these limitations in performance information made it difficult to conduct a full comparison of planned versus actual activities or outputs to assess program implementation.
• Reporting and briefing documents, however, indicate that the Program was implemented as planned for CCCma while CCCSN and research on hazards and extremes experienced a few delays. Program documents for CCCma demonstrate that work on developing Canada's new global climate model was completed and a new model for regional climate predictions was developed and being tested, as planned. While most activities were on track for CCCSN and Hazards in 2009‑2010, CCCSN experienced delays in developing new statistical downscaling models and the hazards component was late in developing new models or methodologies for extreme event analyses. "Staffing issues" were identified in mid-year reviews to senior management and by program staff as the source of these delays. According to program staff as well as some external partners, CCCSN and the hazards and extremes component were under-resourced to carry out their activities.
• Interviewees generally expressed frustration with departmental processes that created barriers or challenges in carrying out their planned activities within the Program.
  • Funding allocation: Program staff noted that delays in receiving Adaptation Theme funding created additional difficulties for the Program to deliver on all of its objectives.^[44] Staffing actions did not proceed as quickly as planned^[45]. As a result, there was slow progress on improvements to the Canadian global climate model for CCCma, with no progress on regional modelling. Moreover, AIRS could not complete its development of new statistical models and tools and the regional nodes were not in place as planned. Some program staff expressed the desire to have A-Base stability for AIRS to avoid similar financial pressures in the future.
  • Staffing processes: The Program experienced some difficulty in hiring the scientists required to conduct adaptation and impacts research. This was reported by program staff as being partly due to the limited supply of qualified candidates and partly due to delays in the staffing process. When a qualified scientist was identified for the position, the candidate accepted another offer, in some cases, while waiting for the staffing action to be finalized. Program staff also expressed concern that applying generalized skill requirements typically used to search for qualified candidates created challenges when staffing for the specialized skills and experience required for the job. The hazards and extremes component, which had the largest number of available positions, was able to staff most of their positions but only on contracts that will terminate once funding ends in March 2011. Program staff note that this lack of permanence has created anxiety among these scientists and many are looking for employment opportunities elsewhere.
• These issues have left the hazards and extremes component of the Program under-resourced compared to what had been planned, which could impact the Program's progress towards achieving expected outcomes. Because Adaptation funds were largely directed towards hazards and extremes research, this component of the Improved Climate Change Scenarios Program were more affected compared to CCCma and CCCSN. To risk manage, AIRS has used casual and contract arrangements to hire and retain personnel to continue work on Program deliverables. From September 2008 through January 2010, 13 positions were staffed on term contracts that are all scheduled to expire by March 2011.
• Interviewees identify a few areas for improvement, primarily concerning client service and program delivery:
  • Reach of training sessions: Interviewees felt that the number of people that receive CCCSN training is small compared to the number of professionals who need to be able to use program information and data. It was suggested that using a shorter, more introductory course for the training (similar to the training provided through Projections in Practice Program (PiP) in the UK) would broaden the reach of these training sessions.
  • Accessibility of website information for clients and partners without climate science expertise: Some interviewees felt that awareness generally increases following training sessions but workshop participants do not necessarily know how to use the information fully. For those without the benefit of CCCSN training sessions, the websites are difficult to navigate and a more user-friendly interface would help target users overcome difficulties in accessing the complex, highly scientific data.
  • Resources for client support, in general: External partners felt that, while appreciated, existing resources for client support are not sufficient. Although they acknowledged the challenges in providing assistance to clients, interviewees felt that they would benefit from increased access to the scientific expertise among program staff to supplement the information they find on program websites.

Top of Page

4.2.3 Demonstration of Efficiency and Economy (Issue 5: Demonstration of Efficiency)

7. Is the Program undertaking its activities and delivering its outputs at the lowest possible cost? How could the cost-efficiency of the Program's activities be improved? Are there alternative, more cost-efficient means of achieving program outputs?

• Indicator(s):
  • Cost analysis of resource utilization in relation to the production of outputs
  • Opinions of stakeholders on the cost-efficiency of delivering program outputs and how it could be improved
• Methods:
  • Document review
  • Key informant interviews
  • Review of financial information
• Rating: Progress made, attention needed

Evidence demonstrates that the Program is employing cost-efficient means to achieve its outputs but financial information was not available to assess cost-efficiency.

• The evaluation was unable to produce a veritable cost-efficiency analysis because sufficiently detailed financial information was not available for each program output across each program component.
• However, the 2008‑2009 Clean Air Agenda Horizontal Performance Report provides higher level financial information up to 2008‑2009. 
  • No funds were requested for 2007‑2008 because program activities were supposed to be funded through the Climate Change Interim Strategy (CCIS) for the first year of the Program.
  • For 2008‑2009, the level of funding spent is approximately 68% less than the amounts allocated.^[46] Interview data and available documentation show that program spending was less than the budgeted amount because funding was received later in the year than expected. This resulted in delays in staffing actions, which resulted in fewer dollars spent in salary and O&M overhead costs for new staff.
• Although it was not possible to draw firm conclusions on the Program's cost efficiency, available information from documentation and key informant interviews point to particular trends and practices on which the Program could focus in future years in order to enhance cost efficiency.
  • Building in-house science capacity: Recruiting new staff allows for knowledge transfer from senior-level scientists and helps the Program to achieve its goals and expected activities for the funded period. Interviews with senior management suggest that, given the critical nature and high relevance of this Program, there could be value added to having permanent staff to carry out the Program's research activities.
  • Partnerships and links with the Canadian climate research community: One of the Program's strengths relating to cost efficiencies are the synergies created with research networks in universities, regional research groups and regional Environment Canada staff, which enhance internal research capacity, reduce duplication of effort and save on some overhead costs. The CCCma, CCCSN and hazards component have taken advantage of collaboration opportunities for research by embedding Environment Canada scientists within Canadian universities in order to harness scientific expertise for achieving the goals of the Program. Interview respondents and internal documents also point to the creation of regional nodes as a means through which the CCCSN has created specific centres of expertise in order to better direct research efforts.
  • CCCSN Training workshops: The workshops, offered by the CCCSN, are designed to teach researchers and other target users how to access and utilize the data available on the CCCSN website. External interviewees and training workshop evaluations have pointed to the high utility of these sessions as well as their capacity to reach many members of the target population at once, saving time and effort among staff in explaining how to use the websites.

8. Is the Program achieving its outcomes at the lowest cost possible? Are there alternative, less costly activities and deliverables that the Program could employ to achieve the same outcomes?

• Indicator(s):
  • Cost analysis of resource utilization in relation to progress toward intended outcomes
  • Presence/absence of less costly alternatives that would achieve the same results
  • Opinions of key informants on whether or not the Program is achieving its outcomes at the lowest possible cost.
• Methods:
  • Document review
  • Key informant interviews
  • Review of financial information
• Rating: Progress made, attention needed

Although sufficiently detailed financial information was not available to assess the economy of the Program, less costly training activities were suggested that may achieve the same outcomes.

• A full cost analysis of resources expended against the achievement of outcomes could not be completed because there was a lack of detailed financial information that clearly links outputs and activities to their associated costs. Mid-year reviews outlined the achievements of the Program components but did not link these achievements with associated costs.
• Most key informants were not able to provide examples of less costly activities and outputs that would achieve the same outcomes. Some key informants, however, suggested some alternatives to CCCSN training that they felt would achieve the same outcomes. For example, an online training system for distance learning could be developed for data and website users or training could be integrated into university curricula in order to increase awareness, availability and access to research, websites and climate models.

[18] Economics of Climate Adaptation (2009). Shaping Climate-Resilient Development: a Framework for Decision-Making, p.10

[19] Intergovernmental Panel on Climate Change (2007): Fourth Assessment of the Intergovernmental Panel on Climate Change, Climate Change 2007: Synthesis Report.

[20] Brooks, Mark, Gagnon-Lebrun, Frédéric, Harvey, Hélène and Sauvé, Claude (March 2009). Prioritizing Climate Change Risks and Actions on Adaptation: A Review of Selected Institutions, Tools and Approaches. Policy Research Initiative, Ottawa, ON.

[21] Lemmen, D.S., Warren, F.J., Lacroix, J., and Bush, E., editors (2008): From Impacts to Adaptation: Canada in a Changing Climate 2007; Government of Canada, Ottawa, ON, p.3.

[22] Office of the Auditor General of Canada (2006). The 2006 Report of the Commissioner of the Environment and Sustainable Development, Chapter 2: Adapting to the Impacts of Climate Change.

[23] Brooks, Mark, Gagnon-Lebrun, Frédéric, Harvey, Hélène and Sauvé, Claude (March 2009). Prioritizing Climate Change Risks and Actions on Adaptation: A Review of Selected Institutions, Tools and Approaches. Policy Research Initiative, Ottawa, ON.

[24] Office of the Auditor General of Canada (2006). The 2006 Report of the Commissioner of the Environment and Sustainable Development, Chapter 2: Adapting to the Impacts of Climate Change.

[25] Environment Canada's 2B5 Adaptation and Impacts Science Plan: An Adaptation and Impacts Science Strategy 2009-2012, the division/section's four-year work plan that addresses the impacts and adaptation science component of the broader departmental plan, Environment Canada's Science Plan: A Strategy for Environment Canada's Science. A similar science plan was not evident for climate modelling activities.

[26] Commissioner of the Environment and Sustainable Development (CESD). Report of the Commissioner of the Environment and Sustainable Development to the House of Commons: Chapter 2: Adapting to the Impacts of Climate Change 2006, p. 7.

[27] Climatic design values refer to measures of climate conditions that have been analyzed and provided for national codes and standards that govern the design requirements for infrastructure. Climatic design values include quantities like the rainfall or weight of snowpack conditions or percentile cold, hot or humid temperature or humidity conditions that are typically derived from historical climate data.

[28] National Round Table on the Environment and the Economy. True North: Adapting Infrastructure to Climate Change in Northern Canada. 2009, p. 85-86.

[29] This model is being developed in collaboration with the Department of Fisheries and Oceans.

[30] Research was considered to be within the scope of this evaluation if the publication date was between 2007 and 2009. Given the delays in the publication process, however, these publications were likely based on research conducted prior to 2007.

[31] Gleckler, P. J., Taylor, K. E., Doutriaux, C. (2008). Performance metrics for climate models, Journal of Geophysical Research, 113. P. 1-20.

[32] Williams, K. D., Webb, M. J. (2009). A quantitative performance assessment of cloud regimes in climate models. Climate  Dynamics, 33-1. p. 141‑157. Retrieved February 26, 2010 from http://www.springerlink.com/content/t4612gv626q2hn44/

[33] This is defined as those having a couple of courses related to meteorology. Even among this 1200, many are likely not to have sufficient expertise and experience to meet the requirements of the job for hazards and extremes research.

[34] This international search poses its own challenges since the position must be sufficiently appealing to draw qualified candidates to Canada and staffing actions must follow HR policy.

[35] It is not clear from the evidence why this decline occurred but the number of data downloads from the CCCma website are likely conservative estimates as they do not take into account the CCCma climate model data that are available for downloading from other international websites, which were out of scope for this evaluation.

[36] Participants are typically scientists who use adaptation science in their own work including, for example, academics, university students, officers from research organizations and other federal departments, municipal planners and conservation authorities, provincial partners and community officers.

[37] Workshops were held in Toronto, Regina, Fredericton and Whitehorse with an average of 22 participants per workshop.

[38] Questions are answered individually by one program officer in the midst of other responsibilities.

[39] Boettcher Crawford, Erica (2009). Bridging Science and Policy for Community Climate Change Adaptation: Is Climate Science "Usable" for Local Practitioners?

[40] Types of climate information included: Risk or vulnerability assessment information, weather or seasonal climate forecasts, projections of specific climatic changes, regional or local climate projections, national or global climate projections, case studies from other jurisdictions, costing of adaptation options, information on best practices and guidebooks on adaptation to climate change. (n=31 communities)

[41] Boettcher Crawford, Erica. Bridging Science and Policy for Community Climate Change Adaptation: Is Climate Science "Usable" for Local Practitioners?, April 30 2009, p. 11.

[42] Toronto Region Conservation Authority (TRCA). Meeting the Challenge of Climate Change: TRCA Action Plan for the Living City, p. 13.

[43] Performance indicators like "publication of new scientific results," "research and analysis to enhance methodologies" or "research and analysis to develop scenarios tools for climate extremes," for example, are not appropriately defined or bounded to provide useful measures of achievement of outputs and intended outcomes.

[44] The Program was already experiencing financial pressures from the decision to redirect Climate Change Interim Strategy (CCIS) funds away from ASTD activities in 2007-08 and reallocate ASTD funding to other parts of the department in 2008-09 to offset the impact of the sunsetting of CCIS. Adaptation funding was intended to be incremental but CCCma and AIRS activities relied almost completely on this funding during this time.

[45] More than half of this funding was intended to be used to hire new scientists.

[46] According to the Clean Air Agenda Horizontal Performance Report 2008-2009, planned spending was $5,210,730 while the actual amount spent totalled $3,567,788. (Figures for other years were not available at time of reporting.) This report also notes that $1,220,000 of this unused funding was to be carried forward to the following two years of the program.

Previous Page | Table of Contents | Next Page