Home >
Proactive Disclosure >
Audits and Evaluations >
A&E Reports >
2009-2010
> Evaluation of the Habitat Stewardship Program for Species at Risk
Evaluation of the Habitat Stewardship Program for Species at Risk
Previous page | ToC | Next page
4.0 Findings
Below are the findings of this evaluation presented by evaluation issue (relevance, success, cost effectiveness, design and delivery)7 and by the related evaluation questions. The findings at the overall issue level are presented first, followed by the findings for each evaluation question.
A rating is also provided for each evaluation question. The ratings are based on a judgment of whether the findings indicate that
- the intended outcomes or goals have been achieved or met–labelled as Achieved;
- considerable progress has been made to meet the intended outcomes or goals, but attention is still needed–labelled as Progress Made, Attention Needed; or
- Little progress has been made to meet the intended outcomes or goals and attention is needed on a priority basis–labelled as Little Progress, Priority for Attention.
- The NA symbol identifies items where a rating is not applicable.
- The ~ symbol identifies outcomes achievement ratings that are based solely on subjective evidence.
A summary of ratings for the evaluation issues and questions is presented in Annex 5.
Except where specifically mentioned, no notable differences were found in findings pertaining to terrestrial and aquatic species. Unless otherwise specified, interview responses were common across Environment Canada, DFO and PCA respondents.
Evaluation Issue: Relevance |
Overall Findings: Numerous sources, including documentation, key informant interviews and survey findings, confirm that the federal government has a legitimate and necessary role in the administration and management of the HSP. Further evidence demonstrates that the HSP is complementary to other funding programs at the federal and provincial levels and that the federal government is uniquely situated to carry out this program. No other jurisdictions or organizations were identified as having the mandate, capacity or interest to implement a similar funding program in the absence of the HSP.
Various lines of evidence point to a strong connection of the HSP with environmental needs. The HSP helps to fulfill the objectives of the SARA, especially with respect to influencing habitat stewardship on non-federal lands, taking regional priorities and needs into consideration. Key informants viewed the HSP as playing a critical role in the recovery of species at risk through the engagement and encouragement of the general public to become involved in habitat stewardship. |
Evaluation Issue: Relevance |
Indicator(s) |
Methods |
Rating |
1. Is there a legitimate and necessary role for the federal government in the HSP? |
- Demonstration of a clear HSP mandate that is aligned with federal government jurisdiction
- Extent to which HSP’s goal and objectives correspond to Environment Canada/DFO/Parks strategic directions, federal government priorities and the National Strategy for the Protection of Species at Risk
- Presence/absence of other programs that complement or duplicate the objectives and/or activities of HSP
|
- Document review
- Interviews
- Survey
|
Achieved |
Numerous sources, including documentation, key informant interviews and survey findings, confirm that the federal government has a legitimate and necessary role in the administration and management of the HSP. Further evidence demonstrates that the HSP is complementary to other funding programs at the federal and provincial levels and that the federal government is uniquely situated to carry out this program. No other jurisdictions or organizations were identified as having the mandate, capacity or interest to implement a similar funding program in the absence of the HSP.
- Since the program was announced in the 2000 Budget, various documents demonstrate that the HSP has a clear mandate that is aligned with federal government jurisdiction. According to official documents, the overall goal of the HSP is to contribute to the recovery of species that are endangered, threatened and of concern, and to prevent other species from becoming a conservation concern, by engaging Canadians in conservation actions to benefit wildlife. In the Cooperative Management Framework for the Strategy for the Protection of Species at Risk, the Minister of the Environment is clearly mandated to preside over the coordination of the federal species-at-risk strategy, including the administration of the HSP.
- Environment Canada’s 2007–2008 Report on Plans and Priorities places the HSP under the departmental program activity “Biodiversity is Conserved and Protected” and the strategic results “Wildlife is conserved and protected and Canadians adopt approaches that ensure the sustainable use and management of natural capital and working landscapes.”
- Although Environment Canada is responsible for administering the program, the HSP is collaboratively managed by Fisheries and Oceans Canada and the Parks Canada Agency. According to Fisheries and Oceans Canada’s 2007–2008 Report on Plans and Priorities, supporting the assessment and recovery of species at risk falls under the strategic outcome “Healthy and Productive Aquatic Ecosystems” and the departmental result “Sustainable development and integrated management of resources in or around Canada's aquatic environment through oceans and fish habitat management.”
- In the 2009–2010 Report on Plans and Priorities for the Parks Canada Agency, the protection and recovery of species at risk fall under the program activity “Heritage Resources Conservation,” with the planned result “to improving the ecological integrity of national parks.” PCA key informants commented that many of the HSP projects that the Agency encourages and supports take place on land that is adjacent to national parks, and in this way extends the conservation activities undertaken by the Agency directly.
- Key informants interviewed confirmed that the HSP aligns with the priorities of the federal government on citizen engagement and the adoption of the ecosystems approach. The loss of habitat is recognized as a major threat to terrestrial species at risk, and the HSP program was assessed as a highly appropriate mechanism for SARA implementation. The Act’s implementation is seen as a key responsibility of the Minister of the Environment because of its direct link to the Department’s mandate.
- Survey results also confirm the importance and relevance of the federal government’s role in this domain. Of survey respondents, 98% reported that there is a moderate to very high need (on a five-point scale where 1 means very low need and 5 means very high need) for habitat stewardship relative to other types of interventions regarding species at risk. According to survey respondents, the most effective way for the federal government to support habitat stewardship is through a contribution program administered by a federal department.
Consulted stakeholders reported that the objectives of the HSP were not duplicated in other similar programs.
- The majority of program managers, RIB members and experts interviewed reported that, while there are other similar programs, these do not share the same priorities as the HSP. Other federal programs include the Ecological Gifts Program, the Endangered Species Recovery Fund, the Interdepartmental Recovery Fund and the Aboriginal Fund for Species at Risk. These programs all support the protection of species at risk but use different means. The Ecological Gifts program aims to protect ecologically sensitive lands, directly targeting individual and corporate landowners through income tax benefits. The Endangered Species Recovery Fund is available to the same types of eligible applicants as the HSP but sponsors research and education projects for the protection or recovery of species at risk. The Interdepartmental Recovery Fund supports recovery activities on federal lands and surveys of species at risk under federal jurisdiction but is only eligible to federal departments and agencies. The Aboriginal Fund for Species at Risk is similar to the HSP except that it also funds capacity-building activities and targets Aboriginal lands and organizations. A stewardship project cannot receive funding from more than one federal program at a time.
- In addition to federal programs, most provinces operate a fund or program devoted to funding environmental stewardship or species-at-risk-related projects. Some experts felt that these provincial programs, while being mostly complementary to the HSP, did overlap somewhat with the program’s objectives. However, the inclusion of provincial representatives on the RIBs facilitates exchange of information across provincial programs and the HSP, thereby limiting the potential for overlap and maximizing complementarity.
Consulted stakeholders reported the following significant gaps in the absence of HSP.
- The HSP is one of the key components of the National Strategy for the Protection of Species at Risk as it applies to non-federal lands. As highlighted in a recent literature review and analysis on SAR governance in Canada, the Species at Risk Act, the core piece of legislation for the SAR National Strategy, offers protections targeting areas of federal constitutional jurisdiction. On provincial and private lands, it defers to provincial legislation and voluntary stewardship in the first instance, providing a “safety net” provision whereby the federal government may step in only in the case of provincial failure8.
- The provinces cannot provide the same coverage as the federal government and this would result in gaps in public engagement for species at risk, especially for migratory birds and aquatics species, and decreased leverage of funding.
Evaluation Issue: Relevance |
Indicator(s) |
Methods |
Rating |
2. Is the HSP connected with environmental needs? |
- Demonstration that HSP addresses identified environmental needs regarding species at risk on non-federal lands and waters
|
- Document and literature review
- Interviews
- Survey
- Case studies
|
Achieved |
Various lines of evidence point to a strong connection of the HSP with environmental needs. The HSP helps to fulfill the objectives of the SARA, especially with respect to influencing habitat stewardship on non-federal lands, taking regional priorities and needs into consideration. Key informants viewed the HSP as playing a critical role in the recovery of species at risk through the engagement and encouragement of the general public to become involved in habitat stewardship.
- Over 500 species have been designated “at risk” by the COSEWIC9. The species at risk in Canada that have been identified by COSEWIC are not evenly distributed and are largely concentrated in southern regions.
- According to official documents, the key objectives of the SARA are to prevent wildlife species from being extirpated or becoming extinct; to provide for the recovery of wildlife species that are extirpated, endangered or threatened as a result of human activity; and to manage species of special concern to prevent them from becoming endangered or threatened. The HSP is identified as being a key program that works in support of these objectives, especially with respect to the implementation of recovery activities, protecting the critical habitat and mitigating threats caused by human activities. As the role of the HSP is to create incentives for other jurisdictions as well as non-government and private players to become involved in stewardship activities, it is the main federal funding program for achieving SAR objectives on non-federal lands and waters.
- It must be noted, however, that some DFO key informants consider that the needs of aquatic species that could potentially become a conservation concern are not adequately addressed by the program. Although prevention is stated as one of the program’s goals, current funding priorities dictate that projects focus on SARA-listed species, which effectively excludes projects exclusively targeting species that are at risk of becoming a conservation concern as assessed by COSEWIC, but that have not been listed under SARA. It must be noted that this limitation applies not only to aquatic species but also to all species that are not currently SARA-listed.
- As well, the Habitat Stewardship Program for Species at Risk Annual Report 2004–2005 states that the program addresses environmental needs with regards to critical habitat, defined under the SARA as “the habitat that is necessary for the survival or the recovery of a listed species and that is identified as the species’ critical habitat in a Recovery Strategy or Action Plan.”10 The HSP provides funding for projects that undertake stewardship activities, which could be located on critical habitats. The addressing of this need, however, is limited by the ability of the three departments/agency to identify critical habitats for species at risk. To date, critical habitats for 22 species at risk have been identified (15 under Environment Canada, four under DFO and three under PCA).
- Since each region under the purview of the HSP has varying environmental needs, the five regions develop a regional prospectus/regional priority statement outlining regional priorities in terms of key species at risk and key habitats on an annual basis. According to the National Call Letter (2007–2008) for the HSP, the RIB chairs must provide evidence as to how the projects being recommended to the NSC reflect the priorities of each region as outlined in the regional priority statements.
- An assessment11 of the HSP was commissioned by the program in 2005. The study’s purpose was to assess the impacts of HSP-funded stewardship activities on 12 species at risk, estimating the degree to which the response of the stewardship activities satisfied the prescribed recovery actions and by characterizing or qualifying the link between stewardship activities supported by the HSP and the prescribed recovery activities. The study relied on interviews with members of species-at-risk recovery teams and examined a total of 326 stewardship activities or groups of stewardship activities relating to 10 species, one ecosystem at risk and one multi-species project, covering the years of the program from 2000 to 2004. The study identified the HSP as important to the recovery of species at risk and their habitats and found that, as one of the sole funding programs supporting non-research projects, it is critical for implementing the types of activities outlined in recovery plans and strategies.
- Key informants interviewed across all three departments/agency identified the HSP as a key tool for addressing environmental needs and found that it plays a unique role in influencing stewardship outside of federal lands through engagement instead of regulation, encouraging the general public to locate species at risk on their lands and develop management and/or conservation plans, and enabling important recovery work that otherwise would not have happened.
- Key informants also identified some gaps that would arise if the HSP were not in existence, including
- a heavier reliance on provincial funding on the part of funded groups;
- a loss of knowledge of species at risk on non-federal lands/waters and gaps in habitat protection; and
- decreased implementation of SAR recovery strategies.
- Of the 130 respondents to the survey of funding applicants, 18 organizations had submitted at least one project funding proposal that had been refused by the program between 2004–2005 and 2007–2008. When asked what had happened to their proposed project following refusal of HSP funding, only one replied that the project had been implemented exactly as planned and five said their project had been implemented with reduced or modified activities. The remaining 12 respondents said their project had been either postponed (8) or cancelled (2) or they were unaware of its status (2).
- When asked to rate the need for habitat stewardship based on their knowledge of interventions in support of species at risk, 68% of respondents to the survey of HSP funding applicants identified a very high need while another 25% identified a high need.
Evaluation Issue: Success |
Overall Findings: Key informant testimonies and high-level performance information collected on the program indicate that it is achieving its intended immediate outcomes. However, doubts remain on the accuracy of the performance data collected and the absence of targets and baseline data limit its usefulness. Raising awareness and engagement results were the most frequently mentioned by key informants and survey respondents but no evidence was found of scientific measurement of such outcomes, measurement being limited to the number of individuals reached or participating in the HSP project activities. Beyond these high-level indicators, evidence of success remains anecdotal and performance indicators are not well adapted to report on aquatic projects. No evidence was found that the program is achieving its intermediate outcomes pertaining to increased species-at-risk populations and species at risk being delisted and reduced in number. This may be due to the long-term nature of these outcomes and the numerous other factors (e.g., climate change, biology of aquatic species, etc.) influencing success, thereby limiting attribution. The most influential external factors affecting the overall success of the HSP were the funding approval delays in 2006–2007 and 2007–2008, the limited availability of recovery strategies, action plans and identification of critical habitat, and the uneven capacity of the environmental NGO community. |
Evaluation Issue: Success |
Indicator(s) |
Methods |
Rating |
3. To what extent have the intended immediate outcomes been achieved as a result of HSP? |
- Extent to which species at risk habitats have been enhanced, protected or restored as a result of HSP
- Extent to which threats to species at risk have been reduced as a result of HSP
- Evidence of HSP information activities reaching target groups of Canadians
|
- Document review
- Review of HSP Online Tracking System for performance data
- Interviews
- Survey
- Case studies
|
~ Progress Made,
Attention
Needed |
Program performance data shows that HSP projects are achieving intended habitat protection, restoration, threat mitigation and education results and several examples were reported of such achievements. However, beyond the number of hectares saved through land acquisition, the validity of program performance data is limited by uncertainty about how it was collected and the absence of direct linkages to critical habitat. The program's effective targeting of stewardship activities is hampered by the limited number of recovery strategies, action plans and identification of critical habitat. While the number of participants in HSP training and outreach activities is an appropriate indicator of Canadians being informed about species at risk, no evidence was collected by the program on Canadians' support for conservation except by proxy through the number of people engaged in conservation activities and the number and breadth of organizations applying for HSP funding.
Overall Success Data
- The program collects performance/outcome information from each funded project through its online tracking system. The performance/outcome information collected is directly tied to the program’s expected outcomes. Table 3 below shows the performance/outcome data collected by the program between 2004–2005 and 2007–2008. It must be noted, however, that since no targets were set for the program until 2007–2008, there is no basis to determine whether these outcomes are satisfactory. Specific findings from this table will be discussed in the sections that follow.
- Furthermore, the performance indicators are not well adapted to report on aquatic projects. Many performance indicators, as shown in Table 3, apply mainly for terrestrial projects. Moreover, counting populations of aquatic species at risk is an ongoing issue given the particular challenges entailed in identifying critical habitats for aquatic species; therefore, monitoring and reporting against the performance indicators is particularly challenging for aquatic species.
- Figures 3 and 4 below provide an overview of key results from the survey of HSP funding applicants pertaining to the issue of success. Specific findings from these figures will be discussed in the sections that follow.
Species-at-risk habitats protected
- According to project data collected through the program tracking system, over the past four years (2004–2005 to 2007–2008) the HSP has contributed to the protection of 65 842 hectares of land through legally binding protection measures (direct acquisition, easements, covenants, leases) and 878 209 hectares of land through non-binding protection measures such as verbal or written stewardship agreements with private landowners. On average, 1513 landowners participated in habitat protection activities each year.
- Most regional coordinators, the RIB members and experts interviewed felt that the HSP projects have contributed to protecting critical habitat, although conceding that evidence beyond the above-mentioned high-level indicators is often anecdotal and that precise measurement of these outcomes is difficult.
- Several key informants pointed out that the strongest guarantee of habitat protection success is when land is purchased. Some consider that land acquisition is the only reliable means of protecting habitat and that there are not enough purchases being made under the HSP. A counter-argument presented by a few key informants suggests that land acquisition, while included as one activity of the HSP, is not the most important aspect of the program, and that the spirit of the HSP in encouraging genuine stewardship activities would be diminished if the program focused primarily on this form of habitat protection. It must also be noted that purchase of land cannot be as easily done to protect habitat of aquatic species at risk because such habitats are often not privately owned.
- Several key informants underscored the need for identifying critical habitat in order for habitat protection and other HSP projects to be adequately targeted. Some key informants questioned the validity and usefulness of the data on the number of hectares saved because they are not tied to critical habitat. As of January 26, 2009, critical habitat had been identified for only 21 species at risk (15 terrestrial species under Environment Canada lead; 4 aquatics species under DFO lead; and 3 terrestrial species under PCA lead).
- A few key informants also highlighted the absence of SAR recovery action plans as a factor limiting the effectiveness of the program. For each species at risk listed under the SARA, the federal government is responsible for the creation of a recovery team, which is in turn expected to develop a recovery strategy, which should be followed by the development of a recovery action plan. To date, there is only one SAR action plan. The HSP projects that target a species at risk for which a recovery strategy exists are expected to be in concordance with the recovery strategy. Furthermore, HSP projects were expected to be a key tool for the implementation of recovery action plans.
- A majority of the HSP funding recipients surveyed believe that their HSP projects have contributed to protecting species-at-risk habitats to some extent: 36% of funding recipients who responded to the survey (Stream 1) said to a great extent (4 and 5 on a 5-point scale where 1 meant to no extent and 5 meant to a great extent) and an additional 27% said to a moderate extent (3 on the 5-point scale).
- The case study of the Mont Rougemont project in Quebec serves to illustrate how certain projects funded by the HSP are achieving this outcome. Since the inception of the Mont Rougemont project in 2002, more than 200 hectares of ecologically important forests have been protected through a combination of land donations, acquisitions, agreements with landowners and the creation of one private natural reserve. This protection of species-at-risk habitats goes in conjunction with other activities carried out by the implicated groups, such as habitat restoration and reduction of threats to species at risk.
- An assessment of HSP results for nine Quebec plant species was conducted in 2005 by the Quebec Region Canadian Wildlife Service. In light of results obtained for projects carried out under HSP between 2000 and 2004, the report concluded that HSP-funded activities had significant effects on all nine of the plant species at risk under study. For example, 20 of the 24 sites that were considered viable in Quebec for American ginseng were assisted through HSP projects. HSP projects were 30% to 90% effective with regards to the protection objective (as laid out in the provincial conservation plan) for these sites. The report concluded that the HSP had a significant effect on the conservation of this species in Quebec12.
- One of the key outcomes featured in the case study on the North Atlantic right whale was the collaboration achieved between stakeholders and partners. As a result of this collaboration, previous research data was shared among the different stakeholders, leading to improved knowledge about the right whale. These research data enabled participants to identify the right whale’s habitat and threats to its habitat so that progress could be made towards protecting its habitat and fostering recovery of the species.
Habitats for species at risk enhanced or restored
- According to HSP performance data for 2004–2005 to 2007–2008, the HSP has contributed to improving a total of 157 669 hectares of species-at-risk habitat (an average of 39 417 ha per year), and a total of 1394 km of shoreline. On average, 6530 individuals participate in habitat improvement activities each year.
- Regional coordinators reported that habitat enhancement or restoration activities were not as much a focus of HSP projects as habitat protection (although this was refuted by survey respondents, who reported in equal proportions that their projects had targeted habitat protection (50%) and habitat improvement (47%) activities). Regional coordinators were able to provide a few examples of the success of these activities but agreed that these outcomes are more difficult to measure and attribute to the HSP than habitat protection outcomes.
- Most RIB members felt that this intended outcome was being achieved to some extent based on the number of HSP projects that include enhancement or restoration activities.
- Similarly, a majority of the experts consulted considered the program as having contributed to this outcome, mainly through outreach, inventory work and invasive alien species removal.
- A majority of Stream 1 survey respondents said their projects had contributed to a great extent (38%) or to a moderate extent (26%) to habitat restoration or enhancement.
- The case study of the Eastern Loggerhead Shrike in Ontario provides an example of a project where significant results were yielded in the enhancement or restoration of species-at-risk habitats. This project involves habitat modelling by a biologist who prioritizes habitat enhancement and restoration activities with the objective of linking areas of habitat together to create a habitat corridor. The activities undertaken as part of this project saw, between 2004–2005 and 2007–2008, the removal of vegetation and exotics from 1241 hectares, the planting of vegetation over 702 hectares, and the improvement of 4 km of riparian (riverbank) shoreline.
Threats to species-at-risk individuals and populations are reduced13
- Performance data show that, on average, 28 867 people participate each year in activities related to the reduction of threats to species-at-risk individuals and populations. This has reportedly resulted in 278 844 species-at-risk individuals14 being protected between 2004–2005 and 2007–2008.
- Regional coordinators were unanimous in affirming that the program has contributed to this outcome and were more easily able to provide specific examples, remarking that these results are generally easier to measure than habitat restoration. Typical interventions leading to reduced threats included outreach work (sensitizing target groups or the general population about the existence of species at risk and their particular habitats, pointing out behaviours that threaten species-at-risk habitat, and proposing alternative behaviours that would better protect them), building fences and promoting the use of modified fishing and harvesting gear.
- Likewise, a majority of the RIB members and experts felt that the program was achieving this outcome, pointing to the success of awareness and outreach activities, such as workshops, training and landowner contacts, but were unable to provide concrete evidence of specific threats being removed.
- A majority of Stream 1 survey respondents said their projects had contributed to a great extent (41%) or to a moderate extent (33%) to the reduction of threats.
- For example, the Mont Rougemont project secured the commitment of landowners through 17 signed agreements where project representatives and biologists worked with the landowners to arrange plans that would allow for the protection of species-at‑risk habitats while allowing continued use and access to the land. Other activities that serve to achieve this outcome were the sensitization of recreational users of the mountain (cyclists, hikers and ATV users), the publication of various educational materials and the conduct of various workshops of interest to landowners and the general public. These activities serve to reduce the threats to individual and populations of species at risk on Mont Rougemont through landowner participation and public awareness of how their activities impact these species.
- Another example of reduced threats could be found in the case study on HSP projects targeting the North Atlantic right whale. Ship strikes, entanglements with fishing gear, etc., have been identified as threats to this particular species. As found in the case study, through HSP projects and previous data collected, it was possible to amend the Bay of Fundy and Roseway Basin shipping lanes to reduce the potential for ship collisions with the whales. As such, the overlap between ships and right whales has been reduced by a large percentage. Where the habitat was identified it was possible, through HSP projects, to reduce the threats.
Canadians informed about species at risk and support conservation
- The HSP tracking database indicates that a total of 30 555 777 Canadians were reached by targeted (e.g., workshop) and non-targeted (e.g., media broadcast) HSP activities between 2004–2005 and 2007–2008. Since there is no formal protocol in place for monitoring the measurement and reporting of this type of data in the HSP tracking database, the accuracy of this particularly high number could not be verified.
- All key informant groups (SAR ADM Committee, National Steering Committee and Secretariat, RIB, and experts) reported that the most prevalent outcomes of HSP projects are increased awareness of species at risk by the target and general populations and increased engagement of target individuals in stewardship activities.
- Regional coordinators explained that, most often, HSP activities target specific groups (e.g., landowners, schools in fishing communities) rather than the general population. Some remarked that these targeted outreach activities offered a better guarantee of sustainable results than information delivered to a more general population.
- One example of successful outreach activity mentioned by key informants: Conservation de la nature and the Centre d'information sur l'Environnement de Longueuil have alerted Longueuil (Quebec) citizens about the threat posed by a development project for the habitat of a listed species of frog in the Boisé du Tremblay. Consequently, citizens lobbied their elected representatives at municipal assemblies and succeeded in having the development project stopped.
- According to one regional coordinator, the success of the stewardship approach has generated interest and increased support within communities for the conservancy goals promoted by participating organizations.
- However, an examination of project final reports contained in the HSP tracking system, as well as testimonies from regional coordinators, indicated that the outcomes of these outreach activities are not systematically or scientifically measured and that reports of success are mostly anecdotal. The principal indicator being measured for this outcome is the number of individuals who took part in outreach activities. No example was found, beyond the occasional questionnaire distributed at the end of an outreach activity, of before and after measurement of participants’ level of awareness, knowledge or support for conservation.
- For example, the Limestone Barrens Stewardship project in the Great Northern Peninsula conducted awareness raising with fishermen who used to dry their fishing gear on top of an area where a very rare type of small flower grows, thereby destroying them in the process. While the project proponent reported that the target fishermen had modified their behaviours following the outreach activities, no scientific before-and-after measurement of this outcome was conducted to determine exactly what proportion of the individuals reached have actually changed their behaviour and whether they have maintained their new practices once the project ended.
- Another example of successful outreach activity was provided by the case study of the Nechako White Sturgeon Recovery Initiative. Participants in this initiative were able to promote protection of the white sturgeon to the communities identified in spite of these communities’ initial reluctance to support the conservation activities. The project obtained buy-in from the communities by furthering communications with First Nation (Carrier Sekani Tribal Council) and non- First Nation communities within the Nechako River watershed. The plight of the Nechako White Sturgeon has subsequently become a rallying point for these communities.
- As illustrated in Figure 3 above, according to the funding recipients surveyed, outreach activities were included in their funded projects 85% of the time, (i.e., the most prevalent of all types of HSP-funded activities), while surveys, inventories and monitoring came second at 75% and program planning and development came third at 65%. These three categories of activities are usually complementary to the core activities of threat mitigation (58%), habitat protection (50%) and habitat improvement (47%). HSP project evaluation activities came last at 39%.
- In keeping with key informants, Stream 1 survey respondents rated the achievement of outreach outcomes highest of all intended HSP outcomes. A large majority said their projects had contributed to a great extent (57%) or to a moderate extent (27%) to Canadians being better informed about species at risk. A slightly smaller proportion said that their projects had contributed to a great extent (50%) or to a moderate extent (31%) to increased support of Canadians for conservation. These ratings are illustrated in Figure 4 above.
Increased recognition of stewardship as a conservation tool
- Most regional coordinators also consider that this outcome is being achieved, mainly through increased NGO understanding and capacity to deliver stewardship activities, more diversified range of applicants for HSP funds, and the successful engagement of landowners in recovery or protection activities. According to these key informants, participating landowners welcome the stewardship approach as a positive alternative to regulation enforcement and are prompt to take ownership of the stewardship activities. The history of the SARA’s development also supports this observation. Two initial attempts at passing the legislation failed in large part due to resistance from lobby groups including landowners, resource industries and agricultural interests. The third attempt was finally endorsed by these stakeholder groups and subsequently accepted by the House of Commons in large part due to its more prevalent stewardship approach.15
- A majority of the RIB members and experts interviewed also think that this outcome is being achieved. This is true mostly of scientists, members of NGOs and landowners who have become more aware of what stewardship entails, but less so among the wider population for whom the term “stewardship” remains vague.
- Although “improved scientific, sociological and economic understanding of the role of stewardship as a conservation tool” was identified early in the life of the program as one of its three main objectives16, the 2004 evaluation of the HSP found that regional stakeholders had limited awareness of this program objective and recommended that emphasis be placed on defining the role of stewardship. The evaluation recommended that the RIB members and the NSC put increased emphasis on evaluating projects, understanding the sustainability of achieved results, and identifying and sharing best practices and lessons learned both within and between program regions. While most key informants consulted as part of the current evaluation were able to provide opinions on the achievement of this objective, the HSP performance measurement strategy still does not have an indicator to track this result.
Other Findings
- An independent study17 of the HSP found, through interviews with experts, that the negotiation and protection of lands, with or without acquisition, targeted outreach, development and preparation of programs and activities, and development or modification of technologies were the activities of the program that had the greatest positive impact on species at risk. The study also found that virtually all of the identified HSP-funded activities were directly related to a recovery action. The study estimated that 89% of activities funded under the HSP have a very significant positive impact on species at risk and that over half (58%) of these activities led to longer-term impacts of greater than five years.
Evaluation Issue: Success |
Indicator(s) |
Methods |
Rating |
4. To what extent have the intended intermediate outcomes been achieved as a result of the HSP? |
- Percentage change of select species-at-risk populations attributable to HSP activities
- Number of species listed as at risk targeted by HSP that have been delisted
- Percentage change in the total number of listed species at risk targeted by HSP
- Change in the level of stakeholders engagement in stewardship activities as a result of HSP
- Opinions of stakeholders and tangible examples of achievement of intended intermediate outcomes
|
- Document review
- Review of HSP Online Tracking System for performance data
- Interviews
- Survey
- Case studies
|
Little Progress, Priority for Attention |
No evidence was found that the program is achieving its intermediate outcomes pertaining to increased species-at-risk populations and species at risk being delisted and reduced in number. This is due to the long-term nature of these outcomes and the numerous other factors (e.g., climate change) influencing success, thereby limiting attribution. Some evidence of Canadians being engaged in species-at-risk conservation was found through the number of people engaged in conservation activities and the number and breadth of organizations applying for HSP funding.
Populations of species at risk are increased
- The HSP tracking system does not contain an indicator for species-at-risk population increases. However, it shows that, on average, 292 COSEWIC species were influenced by the program each year between 2004–2005 and 2007–2008, although the tracking system does not specify whether some or all of these species are the same from year to year.
- Opinions of regional coordinators vary as to the extent the program has achieved this outcome. Some said yes for a few species/projects but not for others. Others mentioned that attribution of success to HSP activities is difficult and that such results will occur in the long term and it is therefore too early to tell. Furthermore, several warned against using higher numbers of species-at-risk individuals as an indicator of HSP success, given that such increases may, at least in the short term, only reflect better data on species-at-risk populations rather than real population increases.
- Half of the experts interviewed said they did not know whether species-at-risk populations had increased because it is still too soon to tell, it is difficult to measure species-at-risk populations, and data is not fed back to recovery teams by the projects or the program. The other half said they only saw anecdotal evidence of the achievement of this outcome.
- A minority of Stream 1 survey respondents said their projects had contributed to populations of species at risk being increased (27% said to a great or moderate extent; 31 percent said to a little extent and 41% said they didn’t know). In response to the argument that attribution of outcomes to HSP projects is difficult, it must however be noted that, when asked what percentage of the success of their projects could be attributed to the HSP, two thirds of Stream 1 survey respondents said 50% or more, which represents a high proportion, given the additional funding leveraged for HSP projects.
- Some RIB members interviewed identified the Piping Plover in the Atlantic Region as an example where an increase in population has been noted in recent years since the implementation of various HSP projects to protect plover habitat. However, it was also noted that it is not scientifically possible to attribute the HSP activities to a species population increase in so short a timeframe.
- In certain cases, an increase in species-at-risk populations can take decades before it can be measured. For example, the white sturgeon can reproduce only after an extended period of time. In fact, the spawning age for white sturgeon males is 15 years of age and more than 20 years of age for females18. The success of this HSP initiative will take many years before it can be measured.
Species listed as at risk are delisted
- None of the key informants interviewed believes that this outcome is being achieved by the program, both because such an outcome is expected to occur in the long term (i.e., 20-year span) and because it is difficult to attribute to the HSP, given the influence of external factors such as climate change.
- A few key informants warned that some species may eventually be delisted due to better identification of their habitat and more accurate counting of their population numbers but not necessarily due to stewardship activities.
- This outcome received the overall lowest rating from Stream 1 survey respondents: only 4% said their project had contributed to a great or moderate extent to species being delisted.
- The program does not track data on the number or type of species being listed or delisted and does not report on the achievement of this outcome. The information can however be obtained through other contacts at Environment Canada and on the SAR Registry website. According to the Species at Risk Annual Report 2006–2007, listed species had augmented from 233 to 425 between June 2003 and 200719. A December 2008 COSEWIC report on Canadian wildlife species at risk stated that, for all the species reassessed by COSEWIC, 227 had stayed in the same category, 76 were placed in a higher risk category and 32 were placed in a lower risk category, of which 20 were delisted (not-at-risk category)20.
Total number of species listed as at risk is reduced
- As this outcome is directly tied to the number of species at risk being delisted, the perspective of key informants was the same as the one described above.
- One regional coordinator and several experts added that the number of listed species is steadily increasing as SAR listing efforts continue: “It’s been a steady increase, which is a function of the process; many species are waiting to be assessed.” One respondent stated that we will likely see a curve upwards before the total number of listed species starts declining.
- Many key informants focusing on aquatic projects stated that the HSP needed to provide funding to prevent species that are not already listed but are likely to become a conservation concern, which would support the program’s goal to prevent other species from becoming a conservation concern.
- A large majority of Stream 1 survey respondents either did not know (44%) or thought that their HSP projects had contributed only to a little extent (49%) to the total number of species at risk being reduced.
Canadians are engaged in species-at-risk conservation
- The program uses the same indicators to measure the extent to which Canadians are engaged in species-at-risk conservation as those used to measure the immediate outcome “Canadians are informed about species at risk and support conservation.” The indicators are limited to the number of people engaged in HSP activities and do not specify the type of engagement.
- A majority of key informants across categories believe that the HSP has contributed to the achievement of this objective. Examples provided include the engagement of conservation organizations and volunteers, as measured through the number and diversity of applicants and the participation of landowners and citizens in the HSP project activities.
- The HSP administrative database shows that 311 different individuals or organizations submitted a HSP funding application between 2004–2005 and 2007–2008, 60% of which received funding for several consecutive years and 14% of which received funding for more than one project per year. Among the 829 applications submitted, 655 projects were funded and completed. Of those, 51% came from local/regional organizations, 35% from provincial organizations, 11% from national/federal organizations, 1% from international organizations, and 1% from organizations for which this information is missing. In addition, an average of 135 organizations were involved as partners in implementing HSP activities annually. Respondents to the survey of funding applicants belong to a wide variety of organizations. While most organizations are small (more than half have 1 to 10 employees), 19% have more than 50 employees and one third recruit more than 50 volunteers on an annual basis. More than half of the organizations have an annual budget of less than $500,000, while 15% operate with more than $5 million. Of all the organizations, 20% had been in existence for 10 years or less, 34% for 11 to 20 years, and the remaining organizations for over 20 years. For more than half of these organizations, habitat stewardship was their primary focus.
- Forty-six percent of Stream 1 survey respondents said their projects had contributed to a great extent to this outcome and 32% said to a moderate extent.
Evaluation Issue: Success |
Indicator(s) |
Methods |
Rating |
5. Have there been any unintended (positive or negative) outcomes that can be attributed to the HSP? If so, were any actions taken as a result of these? |
- Presence/absence of unintended outcomes
- Where appropriate, documented management actions and/or lessons learned from unintended outcomes
|
- Document/file review
- Interviews
- Survey
|
NA |
No unintended outcomes were identified through this evaluation.
Evaluation Issue: Success |
Indicator(s) |
Methods |
Rating |
6. Are there any external factors outside of the HSP that influence the success of the program? |
- Evidence of factors outside HSP which have influenced the achievement of intended outcomes
- Where appropriate, documented management actions to address the influence of external factors
|
- Document/file review
- Interviews
|
NA |
Three principal factors affected the success of the program: the funding approval delays encountered in 2006-2007 and 2007-2008, which impacted the projects, the funded organizations and the program's reputation; the limited availability of SAR recovery strategies, recovery action plans and identification of critical habitats, which limits the strategic targeting of HSP projects; and the uneven capacity of environmental groups to produce proposals and implement projects that address HSP priorities.
- The most important factor affecting the success of the program, as emphasized by every key informant consulted and a majority (80%) of Stream 1 survey respondents, was the delays encountered in obtaining final departmental approval of HSP funding for successful funding applicants for the 2006–2007 and 2007–2008 fiscal years21.
- According to the intended schedule22, HSP funding applications are submitted at the end of fall, the RIB committees submit their funding recommendations by January, and the SAR ADM Committee reviews and endorses funding recommendations in March, with a view to seeing funding approved and contribution agreements negotiated in time for projects to start by April–May. A spring start date for HSP projects is by all accounts necessary, given species-at-risk reproductive cycles and other climate considerations for stewardship interventions.
- In 2006–2007 and 2007–2008, all funding contributions, irrespective of the funding amount, needed to be approved internally by Environment Canada before contribution agreements could be negotiated. Such approval was obtained in the middle of summer. For example, in 2007–2008, the ADMs submitted their funding recommendations on April 4, but the final departmental approval was only obtained on July 16.
- Interview and survey results indicate that these funding approval delays have compromised the feasibility and success of more than half of the projects. According to key informants and 80% of Stream 1 survey respondents, the delays had a number of consequences, including modified/reduced scope of project activities (reported by 73% of Stream 1 survey respondents), eliminated activities (41%), late delivery of results (33%), staff and volunteer recruitment/retention difficulties (18%), and administrative/planning difficulties (17%), frustration of HSP applicants, diminished credibility of the HSP with funding partners, and DFO’s wanting to administer its own contributions. Six of the 90 (7%) survey respondents to this question said they had cancelled their project altogether. Recognizing the challenges posed by these types of delays, Environment Canada has recently conducted a departmental grants and contributions “angiogram” to identify opportunities for improvement. However, this exercise did not address the specific challenges faced by the HSP because it targeted contributions with a value greater than $100,000, whereas the average value of HSP contributions between 2004–2005 and 2008–2009 was $54,000. Nevertheless, the program hopes that in this fiscal year the internal Environment Canada approval process will be accelerated.
- The second most important external factor mentioned by several key informants pertains to the availability of SAR recovery strategies, recovery action plans and identification of critical habitats. As of January 26, 2009, there were 97 species (out of a total of 459 species listed as endangered, threatened and of special concern) for which final recovery strategies had been posted on the Species at Risk Public Registry and critical habitat had been identified for only 21 of those species. As previously mentioned, the HSP projects are expected to be better targeted if they are tied to existing recovery strategies and recovery action plans and if they focus on critical habitat. This limitation was attributed by Environment Canada representatives to the scarcity of available resources. Senior management at Environment Canada has indicated that quicker development of recovery strategies and recovery action plans was a key priority of the Department’s vision for moving forward in implementing the SARA.
- Another important external factor is the uneven capacity within the environmental non-governmental organization (ENGO) community to propose and implement quality projects that address the HSP priorities and the dependency of the program on the quality of proposals received. The Atlantic Region was frequently quoted as facing more significant ENGO capacity issues pertaining to stewardship initiatives. Several factors were evoked to explain this challenge, including the overall small number of existing ENGOs, the limited availability of matching funds and the limited availability of capacity-building funding from the public, private and community sectors for environmental groups in the Atlantic Region. Capacity building is a key element of the vision of Environment Canada’s senior management for the next upcoming phase of the SARA’s implementation.
- Other external factors mentioned as having the potential to affect the success of HSP activities include
- availability of complementary funding, particularly for less glamorous or less appreciated/understood species (e.g., Eastern Massasauga Rattlesnake), further compounded by recent cutbacks to provincial matching funds (e.g., Newfoundland and Prince Edward Island);
- new provincial legislation, which could introduce a regulatory approach that would be incompatible with the SARA’s stewardship approach;
- fluctuations in agriculture market conditions (e.g., mad-cow disease and recent increases in grain prices) affecting availability of matching funds and landowners’ willingness to collaborate in stewardship efforts;
- fluctuations in the federal government’s relationship with Aboriginal groups, which in turn has an impact on implementation of the HSP projects with Aboriginal involvement;
- lack of expertise or limited capacity among environmental groups to prepare proposals, which in turns influences the quantity of viable proposals submitted to HSP; and
- challenges inherent to aquatic projects, including difficulties identifying habitat, particular biology and the life cycle of aquatic species, weather conditions, geography, need for research, etc.
Evaluation Issue: Cost-Effectiveness |
Overall Findings: All the evidence gathered indicates that the program is being delivered cost-efficiently through its strong leveraging of matching funds (2.83:1 ratio), its low administrative costs ratio (13%), and its directed and rigorous application process, which helps ensure that good proposals are developed and funded. The program's online tracking system is also a very cost-efficient tool for administering the contribution agreements and collecting performance information because it cuts down on data entry and processing time. Possible improvements identified by key informants include eliminating funding delays, increasing staff to help develop more targeted proposals, and funding multi-year agreements to reduce the administrative burden on funding recipients. |
Evaluation Issue: Cost-Effectiveness |
Indicator(s) |
Methods |
Rating |
7. Are the most appropriate and efficient means being used to achieve HSP objectives? How could the efficiency of the HSP activities be improved? |
- Comparison of HSP activities to other similar programs
- Resources leveraged from contributions and their associated impact
- Opinion of key informants on the ability of program elements to achieve intended results, compared to alternative design/delivery approaches
- Opinions of key informants on whether HSP investments are a good use of public funds and whether the cost of producing outputs is as low as possible
- Opinions of key informants on how the efficiency of HSP activities could be improved
- Cost analysis
|
- Document review
- Review of program/ project financial data
- Interviews
- Case studies
- Survey
|
Achieved |
- Cost-effectiveness is ideally measured by calculating the costs of achieving the measured program results and comparing them to the costs of achieving the same results through different means. Given the challenges encountered in identifying reliable results data for the HSP, such a calculation could not be done. Consequently, cost-effectiveness was measured indirectly by examining program efficiency and key stakeholders’ perceptions of the program’s value for money.
- All the evidence gathered indicates that the program is being delivered cost- efficiently and offers good value for money:
- There was strong agreement among regional coordinators and the RIB members that the program offers good value for money through its strong leveraging of matching funds, its low administrative costs, and its directed and rigorous application process, which helps ensure that good proposals are developed and funded.
- In addition, because of its online access by funding recipients, program administrators and the RIB members, the HSP tracking system offers a very cost-effective and efficient means of recording and processing applications and contribution agreements, of collecting performance and outcome data, and of generating reports. Such a tracking system cuts down on data entry and processing time.
Table 4 below shows that the budgeted program’s administrative costs represent 13% of its contributions budget. This ratio is lower than those calculated for similar Environment Canada programs. In comparison, the Invasive Alien Species Program had an average ratio of 13% for the period 2006–2007 to 2008–2009 and the ecoAction Program had a ratio of 39% for 2004–2005 to 2007–2008 (this program was intentionally designed to have a high administrative cost ratio due to its broad mandate, predominantly grassroots proponents, regional delivery structure, client service features and the small average value ($25,000) of its contributions).
-
Table 4. HSP Costs for Environment Canada, DFO and PCA
Fiscal Year |
Administrative Costs (Salaries and O&M)*
($) |
Contributions
($) |
Totals
($) |
Administrative Cost Ratio
(%) |
2004–2005 |
1,200,000 |
9,792,167 |
10,992,167 |
12 |
2005–2006 |
1,200,000 |
9,031,461 |
10,231,461 |
13 |
2006–2007 |
1,200,000 |
8,764,133 |
9,964,133 |
14 |
2007–2008 |
1,200,000 |
9,552,650 |
10,752,650 |
13 |
Total |
4,800,000 |
37,140,411 |
41,940,411 |
13 |
* Complete salary and O&M expenditure data were not available for this evaluation. The budgeted amounts were therefore used for the purpose of calculating the program's administrative cost ratio.
- HSP Secretariat staff reported that, for Environment Canada, administrative expenditures have been consistently lower than budgeted amounts. Table 5 below shows the actual average administrative cost ratio for Environment Canada as 8%. Corresponding expenditure figures were not available from DFO and PCA.23
Table 5. HSP Expenditures (Environment Canada only)
Fiscal Year |
Salaries* ($) |
O&M ($) |
Contributions ($) |
Totals ($) |
Administrative Cost Ratio (%) |
2004–2005 |
662,221 |
231,929 |
9,792,167 |
10,992,167 |
9 |
2005–2006 |
609,437 |
119,097 |
9,031,461 |
10,231,461 |
8 |
2006–2007 |
668,204 |
132,500 |
8,764,133 |
9,964,133 |
9 |
2007–2008 |
666,841 |
62,961 |
9,552,650 |
10,752,650 |
8 |
Total |
2,606,703 |
546,487 |
37,140,411 |
41,940,411 |
8 |
*Salaries figures may include expenditures relating to species-at-risk programs in general and not only the Habitat Stewardship Program.
- Table 6 below illustrates the strong leveraging impact of the program. From 2004–2005 to 2007–2008, it has consistently exceeded its minimum 1:1 leveraging ratio requirement. For each HSP dollar invested, an average of $2.83 was obtained in leveraged funding. However, organizations in the Atlantic Region, as well as smaller organizations, are seen as having more difficulty in leveraging matching funds. The Atlantic Region had an average leveraging ratio of 1.76:1 for the same period.
Table 6. HSP Projects, Funding and Leveraging
Funded Year |
# of Projects |
HSP Funding ($) |
Leveraged funding ($) |
Total ($) |
Ratio |
2004–2005 |
174 |
9,792,167 |
30,573,341 |
40,365,508 |
3.12 |
2005–2006 |
147 |
9,031,461 |
20,249,757 |
29,281,218 |
2.24 |
2006–2007 |
166 |
8,764,133 |
26,777,685 |
35,541,818 |
3.06 |
2007–2008* |
194 |
9,552,650 |
27,405,617 |
36,958,267 |
2.87 |
Total |
681 |
37,140,411 |
105,006,400 |
142,146,811 |
2.83 |
* Approximate data.
- Possible improvements to program efficiencies mentioned by key informants include eliminating funding delays and funding multi-year agreements to reduce the administrative burden on funding recipients (although allowed by the program, such agreements were not feasible over the past few years because of the program’s transition between terms and conditions and funding envelopes).
Evaluation Issue: Design and Delivery |
Overall Findings: Available data show that the HSP has a clear governance structure and that processes pertaining to project proposals, review and selection are transparent and adequate. Progress has been made since the 2004 evaluation to address the procedural issues pertaining to project review, eligibility requirements and transparency of decision making. However, some limitations remain for aquatic species, pertaining to the program's eligibility criteria and funding priorities that seem to be better suited for terrestrial species.
The program has also been successful in its ability to leverage resources and build partnerships. However, the program lacks a coherent, strategic and proactive approach to project selection as well as a foundation of baseline data from which results measurement could take place. Although there was consensus around the need for such an approach, the difficulties in identifying critical habitat for species at risk and the limited number of recovery action plans preclude the clear articulation and implementation of a strategic approach to priority setting. |
Evaluation Issue: Design and Delivery |
Indicator(s) |
Methods |
Rating |
8. Are program activities, processes and governance structures adequate for achieving the expected HSP results? |
- Soundness of logical linkages between program mandate, activities, outputs, and intended outcomes
- Defined program processes, roles, responsibilities and accountability
- Opinions of key informants on the adequacy and effectiveness of program activities, processes and governance structures
|
- Document/file review
- Interviews
- Survey
|
Progress Made,
Attention
Needed |
Program activities, processes and governance structures are logical, defined and adequate. However, changes are required in order to strengthen the soundness of the logic model.
Most funding recipients are satisfied with the transparency, clarity and support received from program representatives during the proposal and implementation phases of their projects. Program outreach and communications were found to be informal yet possibly appropriate to the current mode of operation of the program. However, there was agreement across various categories of key informants that the HSP should adopt a more strategic approach to targeting project funding. One suggested approach would consist of focusing on ecosystems rather than species at risk. The limited identification of critical habitat and other capacity issues are factors that may affect the effectiveness of such an approach.
Program Governance and Appropriateness of Activities
- Documents reviewed, including the Cooperative Management Framework for the Strategy for the Protection of Species at Risk, the 2008 Draft HSP RMAF/RBAF, and the National Guidelines on Proposal Evaluation Criteria, clearly outline the HSP governance structure, administrative processes, and roles and responsibilities.
- The HSP logic model shows sound linkages from activities and outputs to outcomes; however, the link between the immediate and intermediate outcomes is less clear as the intermediate outcomes are very high level.
- Interviews with the National Steering Committee (NSC), HSP Secretariat, regional coordinators and the RIB members revealed that these stakeholders believe that the HSP governance and activities are effective and appropriate for the most part, but that there needs to be a stronger link between activities and individual outcomes, outcome measurement needs to be more clearly defined, and intermediate outcomes need to be reworked because they are too long term. These stakeholders also commented on the high turnover experienced on the RIBs, leading to a decreased understanding of roles and responsibilities. Overall, although the RIBs are managed differently in each region, their governance structure was seen as being clear and effective.
- A majority of DFO key informants suggested that the HSP administration and financial management for aquatic projects should be housed within DFO in order to reduce the risk of HSP funds not being released in time to aquatic projects’ funding recipients and to enable their department to better serve its program stakeholders. Such an approach is currently used for managing the Aboriginal Fund for Species at Risk, where projects are identified and approved jointly by Environment Canada, DFO and PCA managers but the funds for aquatic projects are disbursed by DFO following a department-to-department transfer.
Priority Setting, Project Review and Decision Making
- Documents reviewed outline many of the processes in place for funding priority setting, project review and decision making. The processes for priority setting and criteria for project proposal evaluation are clearly laid out in the National Guidelines on Proposal Evaluation Criteria. The proposals that focus on species listed as endangered, threatened and of special concern, first under COSEWIC and second under provincial species-at-risk listings, were clearly identified as priority proposals in the National Call Letter.
- The regional priority-setting documents for each year under the scope of the evaluation were also reviewed and compared. While most regions (and provinces within one region) listed specific priority ecosystems for project proposals, other regions and provinces only listed general ecosystem priorities and priority species.
- A review of relevant documents also showed that the program had effectively taken steps to address the recommendations of the 2004 evaluation pertaining to project review, eligibility requirements and transparency of decision-making processes through the drafting of new policies and program documents to address these concerns.
- Interviews with the NSC and the HSP Secretariat, regional coordinators and the RIB members indicated that the processes in place for setting priorities, reviewing proposals and making funding decisions are clear and adequate.
- A number of issues were raised regarding some regional program funding allocation criteria and eligibility criteria.
- One issue raised by regional coordinators is the challenge posed by the inclusion of provincial distribution as one of the criteria for the allocation of HSP funding within some regions. For instance, a portion (10%) of the Pacific and Yukon Region funding is reserved for species at risk located in the Yukon Territory. However, this territory has few species listed as the most at risk, compared to other areas of the region, making it difficult for the RIB to approve viable projects for that area. Similar challenges are reported for the Prairie and Northern Region, which encompasses three provinces. These key informants therefore suggested that the HSP adopt solely science-based funding allocation criteria (such as concentration of SAR-listed species) rather than include regional distribution considerations.
- Some informants from DFO mentioned that the priority given to SARA-listed species prevents the program from funding prevention activities for species that are at risk of becoming a conservation concern but that are not listed. For instance, during the period examined by this evaluation, no aquatic projects were approved in the Prairie and Northern Region. Although key informants specifically mentioned many challenges in the northern area for aquatic species, there are very few aquatic species listed as at risk within this region.
- Another issue raised pertains to the eligibility of research activities. Research activities of aquatic species habitats are perceived by some informants as particularly important, although technically difficult, due to knowledge gaps about the exact location of these species. However, while the program funds surveys, inventories, etc. that can help to map terrestrial and aquatic habitat and SAR locations, it does not fund fundamental research projects examining species biology, population genetics, etc. Some informants argued that such research is needed to map critical habitat and to adequately target stewardship interventions.
- Funding applicants interviewed indicated that the processes and support in place for proposal review, eligibility, proposal development, implementation and reporting were transparent and adequate. It was expressed that more site visits could be done and that funding was received late in the season. Reporting requirements were noted as being reasonable.
- Survey findings specify the extent to which funding applicants consider that these processes were clearly defined and effective. While 95% of the survey respondents felt the program eligibility criteria were clear, a smaller majority (87%) indicated that the proposal review criteria were clear. Support received from the program during the proposal and implementation processes was available and was felt to be sufficient and clear by a large majority of respondents. As with the interview responses from funding applicants, a majority of survey respondents (65%) believed their organizations had the capacity to meet reporting requirements; the level of funding was also felt to be sufficient for a majority of respondents (65%).
Program Strategic Direction
- Interviews with senior management revealed a consensus among this group towards a need to have a more strategic approach to priority setting and funding of the HSP projects, reflecting the directed nature of the program. Interviews with regional coordinators and experts also echoed this need to have more proactive and science-based project selection and priority setting.
- While most key informants consulted as part of this evaluation support a more strategic approach to allocating HSP funds, varied opinions were expressed as to what this strategic approach should entail.
- Senior management from all three participating departments/agency favour a shift towards an ecosystems-focused (as opposed to species-at-risk focused) approach, whereby fewer and larger projects would be funded, to be implemented by large multidisciplinary teams that would cover more than one species within a target ecosystem. This approach is consistent with the findings and recommendations of the 2006 evaluation of the Species at Risk Program, which reported that the Canadian Endangered Species Conservation Council has directed the core departments/agency to develop a more strategic ecosystem- and multi-species-based approach. The evaluation recommended that the SAR ADM Committee develop “a comprehensive federal vision and strategy to support the preparation and implementation of action plans for the protection and recovery of species at risk and their habitat”. In response to this recommendation, the core departments/agency committed to leading “the development of a vision aimed at streamlining recovery planning and implementation that encompasses a multi-species and/or ecosystems approach, where appropriate.” When the 2006 evaluation report was drafted, uncertainty remained as to what this decision implied for future implementation of key SAR programs, including the HSP. The evidence collected as part of this evaluation indicates that little progress has been made to develop and implement this vision beyond the initial stakeholder consultation conducted as part of the SARA Ministers’ Roundtable held in December 2006.
- Regional coordinators and experts emphasised the need for better identification of critical habitat to more effectively target HSP funding. One informant pointed out that the ecosystems approach is dependent on the development of ecosystems-based recovery and action plans, activities that are outside the scope of the HSP. The HSP could however help implement such plans, once developed.
- All key informants agreed that recovery strategies and recovery plans were essential tools to guide the targeting of habitat stewardship efforts.
- Strategic targeting of the HSP projects requires complete and accurate information on species at risk and their critical habitats, as is currently being collected and developed through recovery strategies, action plans and mapping activities. Documents reviewed indicate that, for species at risk, 67 recovery strategies are in place while another 198 are either delayed or under review. Of these recovery strategies, only one recovery action plan is in place, and it concerns a species found only in a national park and would thus not be implicated in the HSP.
- As of the end of 2008, the identification of critical habitat had only taken place for 22 species at risk altogether (15 under Environment Canada, 4 under DFO and 3 under PCA), representing a small proportion of the total number of species at risk and posing a challenge to the reorientation of the HSP to a strictly ecosystem-based project selection.
- The need to consider new strategic partners, such as industry groups and municipalities, was a recommendation made in the 2004 evaluation of the HSP. The November 2007 update on the recommendations indicates that progress in this area had not taken place and that projects continued to primarily fund NGOs. Program administrative data for the 2006–2007 fiscal year shows that, of the 1397 organizations that contributed matching funds to HSP projects, 112 (8%) were from local/regional governments and 116 (12%) were from the private sector. Interviews with experts revealed that members of this informant group also sensed the need for the HSP to reach out to non-traditional groups such as industry organizations in order to better target projects towards groups with the largest impact on habitats containing species at risk. Some interviewees, however, reported on unsuccessful attempts to engage members of the forestry and mining industries.
Program Outreach and Communications
- Evidence indicates that the program’s outreach and communications are limited to the information provided on its website and its annual targeted calls for proposals, funding announcements and annual reports.
- Interviews conducted with regional coordinators, RIB members and experts revealed that these groups characterized the program’s approach to outreach and communications as being informal but mostly adequate. Although this approach was seen as limiting the number of organizations that were informed of and could apply to the program, it was generally felt that this was done by necessity (the program could not meet a higher demand) and that all groups that would meet the eligibility criteria were already aware of the program.
- Recovery team chairs also pointed to the need for increased communication between the program and recovery teams.
Evaluation Issue: Design and Delivery |
Indicator(s) |
Methods |
Rating |
9. Is performance data collected against program activities/outcomes? If so, is collected information used to inform senior management/decision makers? |
- Presence/absence of populated performance data system with reliable and timely data
- Evidence of decisions based on performance information
- Extent to which performance measurement activities vary between regions
|
- Document/file review
- Review of HSP Online Tracking System for performance data
- Interviews
- Survey
- Case studies
|
Little Progress, Priority for Attention |
Overall, the HSP performance data collection and reporting system was found to be an organized and potentially efficient system, with funding recipients posting information on key outputs and outcomes directly into the centralized database. Potential issues, however, pertain to the integrity, accuracy and completeness of this data. In addition to the absence of formal monitoring of data collection and reporting activities, performance measurement is limited to very high-level indicators that do not provide sufficiently precise measures of the program's coverage and stewardship of critical habitats. Beyond these high-level indicators, the outcome information collected and reported is mostly anecdotal and the HSP tracking system captures this information in text form, which limits its use for analysis and reporting at the program level. The absence of comparable baseline data was also reported as being a detractor from the utility of the performance information collected and reported. The evaluation did not find evidence illustrating the use of performance information for decision making by senior management.
- According to documents reviewed, there is a performance measurement strategy in place for the HSP that includes data collection, analysis and reporting. The 2008 Draft HSP Results-based Management and Accountability Framework/Risk-based Audit Framework (RMAF/RBAF) sets out the processes for the collection, analysis and reporting of performance information. In the program’s project database, funding recipients must identify key outputs and outcomes achieved, such as the degree of leveraged funding, the area of habitat protected, which method was used, and which species or populations the project addresses.24
- The HSP database, managed by the HSP Secretariat located at Environment Canada, was described by one interviewee as being a departmental model for tracking information. The ability for applicants to post their own documents directly onto the system enables quick processing of applications and timely reporting on progress. However, although regional coordinators review the information being entered in the tracking system and ask follow-up questions when they have doubts about the accuracy of the information, there is no formal monitoring of the data collection activities conducted by the funding recipients.
- Interviews with regional coordinators and RIB members revealed that performance information and data are collected and reported in the HSP tracking system. However, there is agreement among these groups that the indicators developed are insufficient and that the linkage between performance data and the achievement of outcomes is anecdotal and tenuous. The longer-term impacts of the HSP activities on species at risk are therefore unclear with the current performance reporting system.
- Interviewed funding applicants and regional coordinators generally agreed that not enough, if any, documentation of baseline information is collected at the outset of a project. This lack of baseline data makes the performance information collected by organizations of little use; it also does not allow for any comparative analyses to occur between projects.
- No evidence could be found illustrating the use of performance information for decision making by senior management.
- A key weakness of the program lies in its inability to accurately and reliably report on project outcomes. The sophisticated tracking system allows project proponents to enter performance data directly into a database, potentially increasing the program’s cost-efficiency. However, the HSP tracking system presents a number of weaknesses:
- There is no formal mechanism for monitoring the quality and accuracy of the data entered by project proponents, with a few exceptions like the requirement for funding recipients to provide documented proof of habitat legally binding protection measures (e.g., acquisition) or the review of other documentation provided by funding recipients.
- The type of data collection conducted by project proponents varies widely across proponents and project types. Results of biological surveys are entered and often supported by the survey reports themselves, but more specific details on the achievements of a project are often entered in text form and therefore cannot be used for HSP-level reporting on results. Furthermore, information on the longer‑term outcomes of the project (e.g., behaviour or attitude changes of project participants) is not being captured at all or only anecdotally.
- The indicators being tracked by the system are very high level and therefore do not allow any distinction to be made between progress made one year and progress being sustained the following year, thereby introducing the possibility that some of the performance numbers reported year after year correspond to the same hectares of habitat being restored or species-at-risk individuals being saved as part of a sustained, multi-year effort. Similarly, nothing precludes more than one funding recipient reporting having reached the same Canadians with different types of outreach activities, which means that some of the Canadians reached being reported may have been counted twice (this appears to be the only possible explanation for the program’s reported average annual reach of 7.6 million Canadians, which represents roughly 24% of the entire Canadian population25. Also, the indicators being captured do not specify whether the progress made is acceptable for a given species (100 ha saved may represent 100% of the habitat for one species but only 3% of another species’ habitat). While these high-level indicators are useful to give an overall sense of the magnitude of the HSP efforts, more specific performance information needs to be captured in ways other than anecdotal.
Evaluation Issue: Design and Delivery |
Indicator(s) |
Methods |
Rating |
10. What are the best practices and lessons learned from the HSP? |
- Identified learnings and best practices
|
- Document review
- Analyses completed for questions 1-9
- Interviews
- Case studies
|
NA |
Three best practices and no lessons learned were identified as part of this evaluation. Best practices included the leveraging effect of the program, the effective quality control tools and practices for ensuring the best selection and orientation of HSP projects, and the advantages of repeated funding for HSP projects targeting the same species or habitats.
Best Practices
- According to interviews conducted with various stakeholder groups, one of the principal best practices of the HSP is the way it capitalizes on the effective collaboration between funding applicants, other partners and private landowners implicated in HSP projects to leverage resources from other organizations and to have those organizations carry out the work, necessitating little overhead from the Department’s perspective.
- In addition, the experts and technical reviewers interviewed identified a number of technical best practices of the program, such as good selection criteria, the adjustment of proposals as a result of expert review, and the linkage of the project proposal to a recovery team.
- The document review and case studies revealed that repeated funding–defined as the funding of several consecutive proposals targeting the same species or habitat, and not to be confused with multi-year funding, which entails the approval of one funding proposal to be implementing over several consecutive years–was found to have a more positive impact on species at risk than the one-time funding of isolated projects. Repeated funding can allow for repeated visits on the part of the funded organization, which has the effect of creating a trustful relationship between funded groups and landowners and positively influencing attitudes towards species at risk.26 While it may be argued that repeated funding implies that some of the same organizations are being funded year after year, which reduces the number of different funding recipients benefiting from the program, key informants indicated that the organizations that have the capacity to develop good proposals and implement HSP projects are already being engaged.
7 The design and fieldwork for the HSP evaluation was carried out in the 2008-09 fiscal year, prior to implementation of the new Treasury Board Policy on Evaluation (http://www.tbs-sct.gc.ca/pol/doc-eng.aspx?id=15024). The current evaluation reflects those issues outlined in the 2001 evaluation policy that was in effect at the time this evaluation was conducted.
8 Kaitlin Alkema, “Sticks and Carrots: Determining the Best Approach to Species-at-risk Governance in Canada” (Master of Resource and Environmental Management Project Report, School for Resource and Environmental Studies, Faculty of Management, Dalhousie University, December 2008), p. 7.
9 Government of Canada, COSEWIC and the Species at Risk Act, available online at http://www.cosewic.gc.ca/eng/sct6/sct6_6_e.cfm.
10 Environment Canada, Habitat Stewardship Program for Species at Risk Annual Report 2004–2005 (March 2007), p. 13.
11 Marc Thibault, Assessment of the Results of the Habitat Stewardship Program for Species at Risk, p. 5.
12 Environment Canada, Canadian Wildlife Service, Quebec Region, Results assessment of the Habitat Stewardship Program for Species at Risk from 2000 to 2004, for nine Quebec plant species: Draft Report (September 2005).
13 The distinction between individuals and populations refers to whether threats are reduced to members of a species or an entire group of one species living in one geographical area.
14 Refers to both fauna and flora individuals.
15 W. Amos, K. Harrison and G. Hoberg, “Search of a Minimum Winning Coalition: The Politics of Species-at-Risk Legislation in Canada” in K. Beazley and R. Boardman (eds.), Politics of the Wild: Canada and Endangered Species (Don Mills: Oxford University Press, 2001), p. 137–166.
16 Environment Canada, Habitat Stewardship (Contribution) Program for Species at Risk – A Results-based Management Accountability Framework and Risk-based Audit Framework: Final Draft (December 3, 2003).
17 Marc Thibault, Assessment of the Results of the Habitat Stewardship Program for Species at Risk, p. 5, 10.
18Nechako White Sturgeon Recovery Initiative, available online at: http://www.nechakowhitesturgeon.org/sturgeon/about/biology/index.php
19 Government of Canada, Species at Risk Act: Annual Report for 2006 and 2007, p.13.
20 Committee on the Status of Endangered Wildlife in Canada, Canadian Wildlife Species at Risk (December 2008).
21 Although within the purview of Environment Canada's internal financial approvals process, this factor was treated as external because it is outside of the control of the program and of the other participating departments.
22 Environment Canada, Habitat Stewardship (Contribution) Program for Species at Risk - A Results-Based Management Accountability Framework and Risk-Based Audit Framework: Final Draft (December 3, 2003).
23 The PCA expenditures for the HSP are estimated at $100,000–$120,000 per year. These expenditures include time spent by the seven staff members who sit on the RIBs and complete other committee work. Collectively, their time amounts to one FTE per year, and the funding comes from the PCA’s SARA budget.
24 Environment Canada, Habitat Stewardship (Contribution) Program for Species at Risk – A Results-based Management Accountability Framework and Risk-based Audit Framework: Draft, p. 24–25.
25 Statistics Canada, Population and Dwelling Count Highlight Tables, 2006 Census, available online at: http://www12.statcan.ca/census-recensement/2006/dp-pd/hlt/97-550/Index.cfm?Page=INDX&LANG=Eng
26 Marc Thibault, Assessment of the Results of the Habitat Stewardship Program for Species at Risk, p. 12; Mont Rougemont case study.
Previous page | ToC | Next page