Evaluation of the Habitat Stewardship Program for Species at Risk

Previous page | ToC | Next page

4.0 Findings

Below are the findings of this evaluation presented by evaluation issue (relevance, success, cost effectiveness, design and delivery)7 and by the related evaluation questions. The findings at the overall issue level are presented first, followed by the findings for each evaluation question.

A rating is also provided for each evaluation question. The ratings are based on a judgment of whether the findings indicate that

A summary of ratings for the evaluation issues and questions is presented in Annex 5.

Except where specifically mentioned, no notable differences were found in findings pertaining to terrestrial and aquatic species. Unless otherwise specified, interview responses were common across Environment Canada, DFO and PCA respondents.


Top of Page


4.1 Relevance

Evaluation Issue: Relevance
Overall Findings: Numerous sources, including documentation, key informant interviews and survey findings, confirm that the federal government has a legitimate and necessary role in the administration and management of the HSP. Further evidence demonstrates that the HSP is complementary to other funding programs at the federal and provincial levels and that the federal government is uniquely situated to carry out this program. No other jurisdictions or organizations were identified as having the mandate, capacity or interest to implement a similar funding program in the absence of the HSP.

Various lines of evidence point to a strong connection of the HSP with environmental needs. The HSP helps to fulfill the objectives of the SARA, especially with respect to influencing habitat stewardship on non-federal lands, taking regional priorities and needs into consideration. Key informants viewed the HSP as playing a critical role in the recovery of species at risk through the engagement and encouragement of the general public to become involved in habitat stewardship.
Evaluation Issue: Relevance Indicator(s) Methods Rating
1. Is there a legitimate and necessary role for the federal government in the HSP?
  • Demonstration of a clear HSP mandate that is aligned with federal government jurisdiction
  • Extent to which HSP’s goal and objectives correspond to Environment Canada/DFO/Parks strategic directions, federal government priorities and the National Strategy for the Protection of Species at Risk
  • Presence/absence of other programs that complement or duplicate the objectives and/or activities of HSP
  • Document review
  • Interviews
  • Survey
Achieved

Numerous sources, including documentation, key informant interviews and survey findings, confirm that the federal government has a legitimate and necessary role in the administration and management of the HSP. Further evidence demonstrates that the HSP is complementary to other funding programs at the federal and provincial levels and that the federal government is uniquely situated to carry out this program. No other jurisdictions or organizations were identified as having the mandate, capacity or interest to implement a similar funding program in the absence of the HSP.

Consulted stakeholders reported that the objectives of the HSP were not duplicated in other similar programs.

Consulted stakeholders reported the following significant gaps in the absence of HSP.

Evaluation Issue: Relevance Indicator(s) Methods Rating
2. Is the HSP connected with environmental needs?
  • Demonstration that HSP addresses identified environmental needs regarding species at risk on non-federal lands and waters
  • Document and literature review
  • Interviews
  • Survey
  • Case studies

Achieved

Various lines of evidence point to a strong connection of the HSP with environmental needs. The HSP helps to fulfill the objectives of the SARA, especially with respect to influencing habitat stewardship on non-federal lands, taking regional priorities and needs into consideration. Key informants viewed the HSP as playing a critical role in the recovery of species at risk through the engagement and encouragement of the general public to become involved in habitat stewardship.


Top of Page


4.2 Success

Evaluation Issue: Success

Overall Findings: Key informant testimonies and high-level performance information collected on the program indicate that it is achieving its intended immediate outcomes. However, doubts remain on the accuracy of the performance data collected and the absence of targets and baseline data limit its usefulness. Raising awareness and engagement results were the most frequently mentioned by key informants and survey respondents but no evidence was found of scientific measurement of such outcomes, measurement being limited to the number of individuals reached or participating in the HSP project activities. Beyond these high-level indicators, evidence of success remains anecdotal and performance indicators are not well adapted to report on aquatic projects. No evidence was found that the program is achieving its intermediate outcomes pertaining to increased species-at-risk populations and species at risk being delisted and reduced in number. This may be due to the long-term nature of these outcomes and the numerous other factors (e.g., climate change, biology of aquatic species, etc.) influencing success, thereby limiting attribution. The most influential external factors affecting the overall success of the HSP were the funding approval delays in 2006–2007 and 2007–2008, the limited availability of recovery strategies, action plans and identification of critical habitat, and the uneven capacity of the environmental NGO community.

Evaluation Issue: Success Indicator(s) Methods Rating
3. To what extent have the intended immediate outcomes been achieved as a result of HSP?
  • Extent to which species at risk habitats have been enhanced, protected or restored as a result of HSP
  • Extent to which threats to species at risk have been reduced as a result of HSP
  • Evidence of HSP information activities reaching target groups of Canadians
  • Document review
  • Review of HSP Online Tracking System for performance data
  • Interviews
  • Survey
  • Case studies
~ Progress Made,
Attention
Needed

Program performance data shows that HSP projects are achieving intended habitat protection, restoration, threat mitigation and education results and several examples were reported of such achievements. However, beyond the number of hectares saved through land acquisition, the validity of program performance data is limited by uncertainty about how it was collected and the absence of direct linkages to critical habitat. The program's effective targeting of stewardship activities is hampered by the limited number of recovery strategies, action plans and identification of critical habitat. While the number of participants in HSP training and outreach activities is an appropriate indicator of Canadians being informed about species at risk, no evidence was collected by the program on Canadians' support for conservation except by proxy through the number of people engaged in conservation activities and the number and breadth of organizations applying for HSP funding.

Overall Success Data

Table 3. HSP Performance Data: 2004–2005 to 2007–2008

Figure 3

Figure 4


Top of Page


Species-at-risk habitats protected

Habitats for species at risk enhanced or restored

Threats to species-at-risk individuals and populations are reduced13

Canadians informed about species at risk and support conservation

Increased recognition of stewardship as a conservation tool

Other Findings

Evaluation Issue: Success Indicator(s) Methods Rating
4. To what extent have the intended intermediate outcomes been achieved as a result of the HSP?
  • Percentage change of select species-at-risk populations attributable to HSP activities
  • Number of species listed as at risk targeted by HSP that have been delisted
  • Percentage change in the total number of listed species at risk targeted by HSP
  • Change in the level of stakeholders engagement in stewardship activities as a result of HSP
  • Opinions of stakeholders and tangible examples of achievement of intended intermediate outcomes
  • Document review
  • Review of HSP Online Tracking System for performance data
  • Interviews
  • Survey
  • Case studies

Little Progress, Priority for Attention

No evidence was found that the program is achieving its intermediate outcomes pertaining to increased species-at-risk populations and species at risk being delisted and reduced in number. This is due to the long-term nature of these outcomes and the numerous other factors (e.g., climate change) influencing success, thereby limiting attribution. Some evidence of Canadians being engaged in species-at-risk conservation was found through the number of people engaged in conservation activities and the number and breadth of organizations applying for HSP funding.

Populations of species at risk are increased

Species listed as at risk are delisted

Total number of species listed as at risk is reduced

Canadians are engaged in species-at-risk conservation

Evaluation Issue: Success Indicator(s) Methods Rating
5. Have there been any unintended (positive or negative) outcomes that can be attributed to the HSP? If so, were any actions taken as a result of these?
  • Presence/absence of unintended outcomes
  • Where appropriate, documented management actions and/or lessons learned from unintended outcomes
  • Document/file review
  • Interviews
  • Survey
NA

No unintended outcomes were identified through this evaluation.

Evaluation Issue: Success Indicator(s) Methods Rating
6. Are there any external factors outside of the HSP that influence the success of the program?
  • Evidence of factors outside HSP which have influenced the achievement of intended outcomes
  • Where appropriate, documented management actions to address the influence of external factors
  • Document/file review
  • Interviews
NA

Three principal factors affected the success of the program: the funding approval delays encountered in 2006-2007 and 2007-2008, which impacted the projects, the funded organizations and the program's reputation; the limited availability of SAR recovery strategies, recovery action plans and identification of critical habitats, which limits the strategic targeting of HSP projects; and the uneven capacity of environmental groups to produce proposals and implement projects that address HSP priorities.


Top of Page


4.3 Cost-Effectiveness

Evaluation Issue: Cost-Effectiveness
Overall Findings: All the evidence gathered indicates that the program is being delivered cost-efficiently through its strong leveraging of matching funds (2.83:1 ratio), its low administrative costs ratio (13%), and its directed and rigorous application process, which helps ensure that good proposals are developed and funded. The program's online tracking system is also a very cost-efficient tool for administering the contribution agreements and collecting performance information because it cuts down on data entry and processing time. Possible improvements identified by key informants include eliminating funding delays, increasing staff to help develop more targeted proposals, and funding multi-year agreements to reduce the administrative burden on funding recipients.
Evaluation Issue: Cost-Effectiveness Indicator(s) Methods Rating
7. Are the most appropriate and efficient means being used to achieve HSP objectives? How could the efficiency of the HSP activities be improved?
  • Comparison of HSP activities to other similar programs
  • Resources leveraged from contributions and their associated impact
  • Opinion of key informants on the ability of program elements to achieve intended results, compared to alternative design/delivery approaches
  • Opinions of key informants on whether HSP investments are a good use of public funds and whether the cost of producing outputs is as low as possible
  • Opinions of key informants on how the efficiency of HSP activities could be improved
  • Cost analysis
  • Document review
  • Review of program/ project financial data
  • Interviews
  • Case studies
  • Survey
Achieved

* Complete salary and O&M expenditure data were not available for this evaluation. The budgeted amounts were therefore used for the purpose of calculating the program's administrative cost ratio.

Table 5. HSP Expenditures (Environment Canada only)
Fiscal Year Salaries*
($)
O&M
($)
Contributions
($)
Totals
($)
Administrative Cost Ratio
(%)
2004–2005 662,221 231,929 9,792,167 10,992,167 9
2005–2006 609,437 119,097 9,031,461 10,231,461 8
2006–2007 668,204 132,500 8,764,133 9,964,133 9
2007–2008 666,841 62,961 9,552,650 10,752,650 8
Total 2,606,703 546,487 37,140,411 41,940,411 8

*Salaries figures may include expenditures relating to species-at-risk programs in general and not only the Habitat Stewardship Program.

Table 6. HSP Projects, Funding and Leveraging
Funded Year # of Projects HSP Funding
($)
Leveraged funding
($)
Total
($)
Ratio
2004–2005 174 9,792,167 30,573,341 40,365,508 3.12
2005–2006 147 9,031,461 20,249,757 29,281,218 2.24
2006–2007 166 8,764,133 26,777,685 35,541,818 3.06
2007–2008* 194 9,552,650 27,405,617 36,958,267 2.87
Total 681 37,140,411 105,006,400 142,146,811 2.83

* Approximate data.


Top of Page


4.4 Design and Delivery

Evaluation Issue: Design and Delivery

Overall Findings: Available data show that the HSP has a clear governance structure and that processes pertaining to project proposals, review and selection are transparent and adequate. Progress has been made since the 2004 evaluation to address the procedural issues pertaining to project review, eligibility requirements and transparency of decision making. However, some limitations remain for aquatic species, pertaining to the program's eligibility criteria and funding priorities that seem to be better suited for terrestrial species.

The program has also been successful in its ability to leverage resources and build partnerships. However, the program lacks a coherent, strategic and proactive approach to project selection as well as a foundation of baseline data from which results measurement could take place. Although there was consensus around the need for such an approach, the difficulties in identifying critical habitat for species at risk and the limited number of recovery action plans preclude the clear articulation and implementation of a strategic approach to priority setting.

Evaluation Issue: Design and Delivery Indicator(s) Methods Rating
8. Are program activities, processes and governance structures adequate for achieving the expected HSP results?
  • Soundness of logical linkages between program mandate, activities, outputs, and intended outcomes
  • Defined program processes, roles, responsibilities and accountability
  • Opinions of key informants on the adequacy and effectiveness of program activities, processes and governance structures
  • Document/file review
  • Interviews
  • Survey
Progress Made,
Attention
Needed

Program activities, processes and governance structures are logical, defined and adequate. However, changes are required in order to strengthen the soundness of the logic model.

Most funding recipients are satisfied with the transparency, clarity and support received from program representatives during the proposal and implementation phases of their projects. Program outreach and communications were found to be informal yet possibly appropriate to the current mode of operation of the program. However, there was agreement across various categories of key informants that the HSP should adopt a more strategic approach to targeting project funding. One suggested approach would consist of focusing on ecosystems rather than species at risk. The limited identification of critical habitat and other capacity issues are factors that may affect the effectiveness of such an approach.

Program Governance and Appropriateness of Activities

Priority Setting, Project Review and Decision Making

Program Strategic Direction

Program Outreach and Communications

Evaluation Issue: Design and Delivery Indicator(s) Methods Rating
9. Is performance data collected against program activities/outcomes? If so, is collected information used to inform senior management/decision makers?
  • Presence/absence of populated performance data system with reliable and timely data
  • Evidence of decisions based on performance information
  • Extent to which performance measurement activities vary between regions
  • Document/file review
  • Review of HSP Online Tracking System for performance data
  • Interviews
  • Survey
  • Case studies
Little Progress, Priority for Attention

Overall, the HSP performance data collection and reporting system was found to be an organized and potentially efficient system, with funding recipients posting information on key outputs and outcomes directly into the centralized database. Potential issues, however, pertain to the integrity, accuracy and completeness of this data. In addition to the absence of formal monitoring of data collection and reporting activities, performance measurement is limited to very high-level indicators that do not provide sufficiently precise measures of the program's coverage and stewardship of critical habitats. Beyond these high-level indicators, the outcome information collected and reported is mostly anecdotal and the HSP tracking system captures this information in text form, which limits its use for analysis and reporting at the program level. The absence of comparable baseline data was also reported as being a detractor from the utility of the performance information collected and reported. The evaluation did not find evidence illustrating the use of performance information for decision making by senior management.

Evaluation Issue: Design and Delivery Indicator(s) Methods Rating
10. What are the best practices and lessons learned from the HSP?
  • Identified learnings and best practices
  • Document review
  • Analyses completed for questions 1-9
  • Interviews
  • Case studies
NA

Three best practices and no lessons learned were identified as part of this evaluation. Best practices included the leveraging effect of the program, the effective quality control tools and practices for ensuring the best selection and orientation of HSP projects, and the advantages of repeated funding for HSP projects targeting the same species or habitats.

Best Practices


7 The design and fieldwork for the HSP evaluation was carried out in the 2008-09 fiscal year, prior to implementation of the new Treasury Board Policy on Evaluation (http://www.tbs-sct.gc.ca/pol/doc-eng.aspx?id=15024). The current evaluation reflects those issues outlined in the 2001 evaluation policy that was in effect at the time this evaluation was conducted.

8 Kaitlin Alkema, “Sticks and Carrots: Determining the Best Approach to Species-at-risk Governance in Canada” (Master of Resource and Environmental Management Project Report, School for Resource and Environmental Studies, Faculty of Management, Dalhousie University, December 2008), p. 7.

9 Government of Canada, COSEWIC and the Species at Risk Act, available online at http://www.cosewic.gc.ca/eng/sct6/sct6_6_e.cfm.

10 Environment Canada, Habitat Stewardship Program for Species at Risk Annual Report 2004–2005 (March 2007), p. 13.

11 Marc Thibault, Assessment of the Results of the Habitat Stewardship Program for Species at Risk, p. 5.

12 Environment Canada, Canadian Wildlife Service, Quebec Region, Results assessment of the Habitat Stewardship Program for Species at Risk from 2000 to 2004, for nine Quebec plant species: Draft Report (September 2005).

13 The distinction between individuals and populations refers to whether threats are reduced to members of a species or an entire group of one species living in one geographical area.

14 Refers to both fauna and flora individuals.

15 W. Amos, K. Harrison and G. Hoberg, “Search of a Minimum Winning Coalition: The Politics of Species-at-Risk Legislation in Canada” in K. Beazley and R. Boardman (eds.), Politics of the Wild: Canada and Endangered Species (Don Mills: Oxford University Press, 2001), p. 137–166.

16 Environment Canada, Habitat Stewardship (Contribution) Program for Species at Risk – A Results-based Management Accountability Framework and Risk-based Audit Framework: Final Draft (December 3, 2003).

17 Marc Thibault, Assessment of the Results of the Habitat Stewardship Program for Species at Risk, p. 5, 10.

18Nechako White Sturgeon Recovery Initiative, available online at:  http://www.nechakowhitesturgeon.org/sturgeon/about/biology/index.php

19 Government of Canada, Species at Risk Act: Annual Report for 2006 and 2007, p.13.

20 Committee on the Status of Endangered Wildlife in Canada, Canadian Wildlife Species at Risk (December 2008).

21 Although within the purview of Environment Canada's internal financial approvals process, this factor was treated as external because it is outside of the control of the program and of the other participating departments.

22 Environment Canada, Habitat Stewardship (Contribution) Program for Species at Risk - A Results-Based Management Accountability Framework and Risk-Based Audit Framework: Final Draft (December 3, 2003).

23 The PCA expenditures for the HSP are estimated at $100,000$120,000 per year. These expenditures include time spent by the seven staff members who sit on the RIBs and complete other committee work. Collectively, their time amounts to one FTE per year, and the funding comes from the PCA’s SARA budget.

24 Environment Canada, Habitat Stewardship (Contribution) Program for Species at Risk – A Results-based Management Accountability Framework and Risk-based Audit Framework: Draft, p. 24–25.

25 Statistics Canada, Population and Dwelling Count Highlight Tables, 2006 Census, available online at:  http://www12.statcan.ca/census-recensement/2006/dp-pd/hlt/97-550/Index.cfm?Page=INDX&LANG=Eng

26 Marc Thibault, Assessment of the Results of the Habitat Stewardship Program for Species at Risk, p. 12; Mont Rougemont case study.

Previous page | ToC | Next page