This report presents the results of the evaluation, between October 2008 and March 2009, of the Habitat Stewardship Program (HSP) for Species at Risk led by Environment Canada’s Audit and Evaluation Branch, with the participation of Fisheries and Oceans Canada (DFO) and Parks Canada Agency (PCA) evaluators. The program was selected for evaluation to support decision making, since the program’s terms and conditions were expiring at the end of 2008–2009. Preliminary evaluation findings were provided to program management during the 2008–2009 fiscal year.
The evaluation examined the overall effectiveness of the HSP, focusing on issues of relevance, success, cost-effectiveness and design and delivery. The scope of the evaluation covers all five regions of Environment Canada, focusing on the fiscal years from 2004–2005 and 2007–2008. The findings of this evaluation relied on multiple lines of evidence, including a review of relevant documentation, key informant interviews with 50 various stakeholders, an online survey of HSP funding applicants, five case studies and a review of program performance data.
The overall goal of the HSP, as one of the three pillars of Canada’s National Strategy for the Protection of Species at Risk, is to provide financial support for stewardship activities that contribute to the survival and the recovery of designated flora and fauna and their habitats. The HSP is a collaborative effort between Environment Canada, DFO and PCA; however, overall responsibility and accountability for HSP contributions lie with the Minister of the Environment. The program is implemented at the regional level, where regional coordinators work in collaboration with the regional implementation boards (RIBs) in making recommendations on project funding. The HSP Secretariat resides within Environment Canada, and the interdepartmental SAR Assistant Deputy Minister (ADM) Committee, further supported by an operations management committee at the Director‑General level, is responsible for the overall governance of the HSP and final decisions regarding project approval and funding. Eligible recipients of HSP funding include not-for-profit organizations, Aboriginal organizations, educational institutions, community associations and local groups, private individuals and companies, and provincial, municipal and local governments.
Out of the Species at Risk Program budget, $10 million was initially allocated to the HSP from 2003–2004 to 2007–2008 for funding stewardship contribution agreements. Subsequently, however, the allocation was reduced to $9 million in Budget 2005 following expenditure review. An additional $4 million was allocated annually for the 2007–2008 to 2011–2012 fiscal years period. The annual budget allocated to manage the HSP is approximately $1.2 million.
This evaluation of the HSP revealed that the program is highly relevant to federal government roles and priorities, and is also a particularly appropriate means of federal government intervention for addressing the needs of species at risk on non-federal lands because of its collaborative approach. While several similar programs co-exist with the HSP, none duplicates the program’s targeted activities.
The HSP was designed as a directed program to focus contribution funds on key program priorities, in collaboration with communities of interest where known capacity to deliver conservation exists. However, evidence has shown that the program has been ineffective in fulfilling this aspect of its mandate, since funding decisions have been mostly reactive to the proposals being submitted and, according to key informants from all categories, the program needs to adopt a more strategic approach to allocating HSP funds because of the limited pool of funds available for HSP projects.
According to the evidence collected, the program’s difficulties in fulfilling its mandate as a directed program can be attributed to two main factors. First and foremost, achievement of HSP intended outcomes is closely linked to the implementation of the wider federal SAR Program. Various sources showed that the SAR Program faces challenges that must first be resolved before the HSP can possibly fulfill its mandate. Such challenges include slow progress in the identification of critical habitats and the development of SAR recovery and action plans. Such guides are needed to help focus HSP project activities. These activities are all beyond the current scope of the HSP; yet, slow progress in these aspects of the SAR Program affects the HSP’s ability to strategically focus its funding. In response to a recommendation of the 2006 evaluation of the SAR Program, the core departments/agency had committed to developing a comprehensive federal vision and strategy to support the preparation and implementation of SAR recovery action plans. Recent follow-up has shown that little progress has been made to articulate and implement this vision and to determine its implications for SAR programs, including the HSP.
Another factor impeding the program’s ability to strategically focus its funding is the uneven capacity of the environmental NGO community to develop and implement quality project proposals that address HSP priorities, combined with the program’s limited capacity to reach out to new potential funding applicants and strategic partners that might be able to respond to the program’s priorities. Various stakeholders believe that regional staffs have a central role to play to develop NGO capacity and to articulate projects that might interest new strategic partners. To date, most Environment Canada regional coordinators and their DFO and PCA regional counterparts have had difficulty engaging in this role due to limited resources being assigned to HSP delivery.
In its current form, the program is well managed and administered. The program governance structure is clear and effective, and the RIBs offer a good model of consensus decision making involving multiple jurisdictions.
Overall, existing program resources are sufficient to support processing of the project applications, decision making and administration of the contribution agreements. They are insufficient to enable regional staff to develop NGO capacity and to actively monitor project performance and reporting.
Limited information could be gathered on the program’s success in achieving its immediate outcomes because of limitations inherent to the type of indicators being tracked, uncertainty about the accuracy of some of the performance data collected, and the absence of targets and baseline data. Also, no evidence was found that the program is achieving its intermediate outcomes pertaining to increased species-at-risk populations and species at risk being delisted and reduced in number, in part because such outcomes can rarely be attributed solely to project activities and, according to all stakeholders interviewed, because most HSP project impacts on species at risk will take decades to occur. Furthermore, the program design and performance indicators, with their habitat focus, are better suited to address and measure impacts on terrestrial than aquatic species.
According to key informants and survey respondents, the strongest results achieved by the program pertain to the education and engagement of Canadians, followed by habitat protection results.
Delays encountered in 2006–2007 and 2007–2008 in obtaining Environment Canada departmental approval of the project selection were a key external factor mentioned by every informant as having affected the program’s effectiveness. Moreover, a majority of DFO key informants suggested that the HSP administration and financial management for aquatic projects should be housed in Fisheries and Oceans Canada.
The following recommendations are based on the findings and conclusions of the evaluation. The evaluation recommendations are directed to the SAR ADM Committee in light of its responsibility for the overall management of SARA implementation.
Recommendation 1a: Evidence shows that the program is not adequately fulfilling its mandate as a directed program. It has been largely ineffective at strategically identifying priorities on which to focus program funding. This limitation is attributed first and foremost to the fact that the achievement of HSP intended outcomes is closely linked to the implementation of the wider federal SAR Program. The SAR Program faces challenges that must first be resolved before the HSP can possibly fulfill its mandate. Such challenges include slow progress in the identification of critical habitats and the development of SAR recovery and action plans. Such guides are needed to help focus HSP project activities. These activities are all beyond the current scope of the HSP; yet, slow progress in these aspects of the SAR Program affects the program’s ability to strategically focus HSP funding. In response to a recommendation of the 2006 evaluation of the SAR Program, the core departments/agency had committed to developing a comprehensive federal vision and strategy to support the preparation and implementation of SAR recovery action plans. Recent follow-up has shown that little progress has been made to articulate and implement this vision and to determine its implications for SAR programs, including the HSP. In light of these considerations, it is important that the core departments/agency increase efforts to identify critical habitats and to implement the management response to the 2006 SAR evaluation recommendation pertaining to the development of a comprehensive federal vision and strategy to support the preparation and implementation of recovery action plans. It is therefore recommended that members of the SAR ADM Committee develop guidance documents to ensure a more timely identification of critical habitats and development of action plans for the protection and recovery of species at risk and their habitats.
Recommendation 1b: It is recommended that members of the SAR ADM Committee clearly articulate the expected role of the HSP in the implementation of the wider federal SAR Program and revise the HSP funding allocation criteria and formula accordingly.
Recommendation 2: Another factor impeding the HSP’s ability to strategically focus its funding is the uneven capacity of the environmental NGO community to develop and implement quality project proposals that address HSP priorities, combined with the program’s limited capacity to reach out to new potential funding applicants and strategic partners that might be able to respond to the program’s priorities. Various stakeholders believe that regional staff have a central role to play in developing NGO capacity and in articulating projects that might interest new strategic partners. To date, most Environment Canada regional coordinators and their DFO and PCA regional counterparts have had difficulty engaging in this role due to limited resources being assigned to HSP delivery. Given that the expected role for the HSP in supporting the implementation of the SAR Program has yet to be fully articulated, it is premature to recommend the development of a precise plan for addressing these limitations. It is recommended that the SAR ADM Committee: first, conduct an assessment of the capacity required to adequately fulfill the HSP's mandate; and, secondly, ensure that the program has, or has access to, this required capacity.
Recommendation 3: Evidence collected as part of this evaluation illustrated the need to articulate intermediate outcomes that can be measured and reported on in a shorter time frame than the 20-year span HSP stakeholders have unanimously identified as being required for the achievement of the current intermediate outcomes (as found in the 2003 and 2008 HSP logic model) pertaining to species at risk. When evaluated against the 2003 and 2008 logic models, the program is unable to demonstrate the achievement of results beyond immediate outcomes. It is recommended that members of the SAR ADM Committee oversee revisions to the HSP logic model in view of identifying intermediate outcomes that can be measured and attributed to the program within a five-year timeframe.
Recommendation 4a: A key weakness of the HSP lies in its limited ability to reliably demonstrate the achievement of its intended outcomes. This is in part due to the fact that there is no formal and systematic mechanism in place for monitoring the collection and reporting of performance/outcome data by funding recipients. This absence of formal monitoring undermines confidence in the quality and accuracy of some of the performance/outcome information collected by funding recipients and subsequently reported by the program. While informal monitoring is being conducted by regional staff, they reported that they lack capacity to conduct monitoring site visits. Concurrently, funding recipients expressed the desire to receive more site visits from regional program staff. Furthermore, existing performance indicators are not well adapted to report on aquatic projects. In light of these considerations, it is recommended that members of the SAR ADM Committee oversee the development of a formal and systematic mechanism for monitoring the collection and reporting of performance/outcome data by funding recipients.
Recommendation 4b: Further, it is recommended that members of the SAR ADM Committee oversee improvements to monitoring and reporting of results of aquatic projects through adjustments to the existing program performance indicators.
Recommendation 5a: The most important factor identified as having affected the success of the program was the delays encountered in obtaining final departmental approval of HSP funding for the 2006–2007 and 2007–2008 fiscal years. Final HSP funding approvals are within the purview of Environment Canada’s internal financial approvals processes. These delays impacted the projects, the funded organizations and the program’s reputation. A majority of DFO key informants suggested that this challenge would be in part mitigated if their department could separately administer the funds dedicated to aquatics projects, and that this would allow DFO to better serve its target stakeholders. It is thereby recommended that the SAR ADM Committee consider alternative options to expedite the administration of HSP funds, including the possibility for each participating department/agency to disburse HSP funds to its respective target stakeholders.
Recommendation 5b: Further, it is recommended that the ADM responsible for the HSP at Environment Canada identify ways to avoid delays in future Environment Canada financial approvals processes for HSP projects under the Department’s lead.
The SAR Assistant Deputy Minister (ADM) Committee accepts the evaluation and its recommendations. As the Habitat Stewardship Program (HSP) for Species at Risk is part of the Species at Risk Program, the full implementation of the management response is also subject to considerations and requirements that may emerge from the 5-year Parliamentary Review of the Species at Risk Act (underway) and of the conclusions and recommendations of the next SAR Program Evaluation in 2010-2011.
Recommendation 1a: It is recommended that members of the SAR ADM Committee develop guidance documents to ensure a more timely identification of critical habitats and development of action plans for the protection and recovery of species at risk and their habitats.
The SAR ADM Committee agrees with the recommendation.
Independently of the evaluation of the Habitat Stewardship Program for Species at Risk, the SAR ADM Committee had already agreed to ensure: 1) completion and publication on the SARA Public Registry, by 2009, of the SARA Policy Framework which includes sections on Assessment, Protection, Recovery Planning, Implementation, and Monitoring and Evaluation; 2) completion, by Fall 2009, of a guidance document related to Recovery Strategies Content; 3) completion, by Summer 2010, of guidance documents related to Critical Habitat (Identification, Destruction, Legal Protection and Effective Protection of Critical Habitat) and 4) completion by Fall 2010 of a guidance document related to Action Plans Content. Completion of this work will improve the timeliness of the identification of critical habitat and of the development of recovery action plans.
Recommendation 1b: It is recommended that members of the SAR ADM Committee clearly articulate the expected role of the HSP in the implementation of the wider federal SAR Program and revise the HSP funding allocation criteria and formula accordingly.
The SAR ADM Committee agrees with the recommendation.
The SARA Policy Framework, to be posted on the SARA Public Registry in 2009, includes an overview of the mechanisms, including the HSP, intended to implement recovery strategies, action plans and management plans. Furthermore, the SAR ADM Committee will mandate, in Fall 2009, a working group to examine all the mechanisms in place to implement the SAR Program in order to review and clearly articulate, by Spring 2010, the role and mandate of each of these mechanisms, including the HSP. With the role of the HSP clearly articulated, the funding allocation criteria and formula, will be revised accordingly by Summer 2010.
In the interim, the national call letter for 2010-2011 has been revised to reflect SAR ADM Committee priorities, including directing funds to geographic and threat-based priority areas for species at risk and encouraging geographically conservation planning for SAR recovery.The national call letter was approved by the SAR ADM Committee at the end of August 2009 and the regional funding allocation criteria and project evaluation grid are being revised accordingly.
Recommendation 2: It is recommended that the SAR ADM Committee first, conduct an assessment of the capacity required to adequately fulfill the HSP's mandate and, secondly, ensure that the program has, or has access to, this required capacity.
The SAR ADM Committee agrees with the recommendation.
As part of the articulation, by Spring 2010, of the role and mandate of each of all the mechanisms in place to implement the SAR Program, the SAR ADM Committee will carefully assess the resource and capacity implications of the role identified for the HSP and, if necessary, will identify and seek additional resources to enable the HSP to fulfil this new orientation.
The SAR ADM Committee will also oversee improvements to the communication of HSP priorities to program partners and proponents.
Recommendation 3: It is recommended that members of the SAR ADM Committee oversee revisions to the HSP logic model in view of identifying intermediate outcomes that can be measured and attributed to the program within a 5-year timeframe.
The SAR ADM Committee agrees with the recommendation.
The SAR Funding Programs (AFSAR, HSP and IRF) are developing a logic model and key performance indicators and the HSP Performance Measurement Strategy is being revised by the National Steering Committee accordingly. The SAR ADM Committee will ensure that the intermediate outcomes for HSP are realistic, appropriate, and can be measured and attributed to the program within a 5-year timeframe. The HSP Performance Measurement Strategy will be presented to the SAR ADM Committee for approval by Winter 2009/10.
Recommendation 4a: It is recommended that members of the SAR ADM Committee oversee the development of a formal and systematic mechanism for monitoring the collection and reporting of performance/outcome data by funding recipients.
The SAR ADM Committee agrees with the recommendation.
The SAR Funding Programs (AFSAR, HSP and IRF) are developing a logic model and key performance indicators, and the HSP Performance Measurement Strategy is being revised by the National Steering Committee accordingly. The HSP Performance Measurement Strategy outlines the program logic model and performance measurement plan, including the performance measures, indicators and the supporting data requirements, and data collection and verification strategy. The Performance Measurement Strategy will include a formal and systematic mechanism for monitoring the collection and reporting of performance/ outcome data by funding recipients. The HSP Performance Measurement Strategy will be presented to the SAR ADM Committee for approval by Winter 2009/10.
Recommendation 4b: It is recommended that members of the SAR ADM Committee oversee improvements to monitoring and reporting of results of aquatic projects through adjustments to the existing program performance indicators.
The SAR ADM Committee agrees with the recommendation.
The SAR Funding Programs (AFSAR, HSP and IRF) are developing a logic model and key performance indicators and the HSP Performance Measurement Strategy is being revised by the National Steering Committee accordingly. The SAR ADM Committee will ensure that specific outcomes and related performance indicators are included for monitoring and reporting the results of aquatic projects. The HSP Performance Measurement Strategy will be presented to the SAR ADM Committee for approval by Winter 2009-2010.
Recommendation 5a: It is recommended that the SAR ADM Committee consider alternative options to expedite the administration of HSP funds, including the possibility for each participating department/agency to disburse HSP funds to its respective target stakeholders.
The SAR ADM Committee agrees with the recommendation to examine this option.
As in the past, the SAR ADMs Committee will consider alternative options, including the possibility for each participating department/agency to disburse HSP funds and administer the contribution agreements with its respective target stakeholders no later than the next renewal of the federal species at risk program in 2011-2012.
Recommendation 5b: It is recommended that the ADM responsible for the HSP at Environment Canada identify ways to avoid delays in future Environment Canada financial approvals processes for HSP projects under the department’s lead.
The EC ADM acknowledges the issue with the timing of the allocation of the EC grant and contribution budget. Recognizing that others in EC have the responsibility for the allocation of departmental budgets, the EC ADM will continue to work within EC toward an allocation that enables HSP contribution agreements to be signed by the beginning of each fiscal year. The EC ADM made a significant contribution to obtaining approval for 2009-2010 HSP contribution budget on April 28, 2009, significantly earlier than the three preceding fiscal years.