Government of Canada
Symbol of the Government of Canada

Common menu bar links

Evaluation of the Habitat Stewardship Program for Species at Risk

Previous page | ToC | Next page

Annex 2
Evaluation Matrix

Evaluation Issues, Questions, Indicators, Data Sources and Data Collection Methods

Evaluation Questions

Statement of
What Should Be Observed

Indicators

Information Sources/ Methods

Relevance
1. Is there a legitimate and necessary role for federal government in HSP?
  • HSP is aligned with identified SAR, departmental and federal priorities
  • Program delivery complements but does not duplicate other programs
  • Demonstration of a clear HSP mandate that is aligned with federal government jurisdiction
  • Extent to which HSP’s goals and objectives correspond to Environment Canada/DFO/PCA strategic directions, federal government priorities and National Strategy for the Protection of Species at Risk
  • Presence/absence of other programs that complement or duplicate the objectives and/or activities of HSP
  • Document review (e.g., legislation/policies; Treasury Board submissions; speeches from the Throne and budgets; annual reports; departmental performance reports (DPRs) and reports on plans and priorities (RPPs))
  • Interviews with NSC members; HSP Secretariat; RIB members; regional coordinators; experts/recovery team members; provincial/territorial representatives
2. Is HSP connected with environmental needs?
  • HSP serves the environmental needs of species at risk
  • Demonstration that HSP addresses identified environmental needs regarding species at risk inhabiting non-federal lands and waters
  • Document and literature review
  • Interviews with NSC members; HSP Secretariat; RIB members; regional coordinators; experts/recovery team members; provincial representatives; funding recipients
  • Funding proponents/recipients survey
  • Case studies
Success/Impact
3. To what extent have the intended immediate outcomes been achieved as a result of HSP?
  • Species-at-risk habitats protected
  • Habitats for species at risk enhanced or restored
  • Threats to species-at-risk individuals and populations reduced
  • Canadians informed about species at risk and support conservation
  • Extent to which species-at-risk habitats have been enhanced, protected or restored as a result of HSP (e.g., increase in the number of ha/km of shoreline protected or improved and number of participating landowners)
  • Extent to which threats to species at risk have been reduced as a result of HSP (e.g., increase in the number of species-at-risk individuals protected)
  • Evidence of HSP information activities reaching target groups of Canadians (e.g., increase in the number of people reached and participants engaged)
  • Document review (e.g. , project final reports, program annual reports, other program documents)
  • Review of HSP Online Tracking System for performance data
  • Interviews with NSC members; HSP Secretariat; RIB members; regional coordinators; experts/recovery team members; provincial representatives; funding recipients
  • Funding proponents/recipients survey
  • Case studies
4. To what extent have the intended intermediate outcomes been achieved as a result of HSP?
  • Populations of species at risk are increased
  • Species listed as at risk are delisted
  • Total number of species listed as at risk is reduced
  • Canadians are engaged in species-at-risk conservation
  • Percentage change of select species-at-risk populations attributable to HSP activities
  • Number of species listed as at risk targeted by HSP that have been delisted
  • Percentage change in the total number of listed species at risk targeted by HSP
  • Change in the level of stakeholders engagement in stewardship activities as a result of HSP (e.g., actions identified in recovery strategies)
  • Opinions of stakeholders and tangible examples of achievement of intended intermediate outcomes
  • Document review
  • Review of HSP Online Tracking System for performance data
  • Interviews with NSC members; HSP Secretariat; RIB members; regional coordinators; experts/recovery team members; provincial representatives; funding recipients
  • Funding proponents/recipients survey
  • Case studies
5. Have there been any unintended (positive or negative) outcomes that can be attributed to HSP? If so, were any actions taken as a result of these?
  • Unintended outcomes are present that can be attributed to the program
  • Where appropriate actions to address unintended outcomes were taken
  • Presence/absence of unintended outcomes
  • Where appropriate, documented management actions and/or lessons learned from unintended outcomes
  • Document/file review (e.g., meeting minutes and presentations; project and annual reports)
  • Interviews with Secretariat; RIB members; regional coordinators; experts/recovery team members; provincial representatives; funding recipients
  • Funding proponents/recipients survey
6. Are there any external factors outside of HSP that influence the success of the program?
  • Factors external to HSP are identified that have influenced the achievement of results
  • Evidence of factors outside HSP that have influenced the achievement of intended outcomes (e.g., partner capacity and cooperation, information about species at risk, other policies and incentives)
  • Where appropriate, documented management actions to address the influence of external factors
  • Document/file review (e.g., planning documents; meeting minutes; project and annual reports; other program documents)
  • Interviews with NSC members; HSP Secretariat; RIB members; regional coordinators; experts/recovery team members; provincial representatives; funding recipients
Cost-Effectiveness
7. Are the most appropriate and efficient means being used to achieve HSP objectives? How could the efficiency of HSP activities be improved?
  • Alternative design/ delivery methods that are less costly/effort-intensive, enable quicker achievement of results, and /or enable achievement of better results
  • Evidence of leveraging of funds from non-federal sources
  • HSP activities show opportunities for efficiency increases
  • Comparison of HSP activities to other similar programs
  • Resources leveraged from contributions and their associated impact
  • Opinion of key informants on the ability of program elements to achieve intended results, compared to alternative design/delivery approaches
  • Opinions of key informants on whether HSP investments are a good use of public funds and whether the cost of producing outputs is as low as possible
  • Opinions of key informants on how the efficiency of HSP activities could be improved
  • Cost analysis
  • Document review (e.g., program design reports; available documents on other programs)
  • Review of program/project financial data
  • Interviews with NSC members; HSP Secretariat; RIB members; regional coordinators; experts/recovery team members; provincial representatives; funding recipients
  • Case studies
  • Funding proponents/recipients survey
Design and Delivery
8. Are program activities, processes and governance structures adequate for achieving expected HSP results?
  • HSP design is aligned with its overall mandate, identified needs and intended program outcomes
  • Program processes are adequate and effective
  • Roles, responsibilities and accountability are clear, adequate and duly implemented
  • Soundness of logical linkages between program mandate, activities, outputs and intended outcomes
  • Defined program processes, roles, responsibilities and accountability
  • Opinions of key informants on the adequacy and effectiveness of program activities, processes and governance structures

Specific processes to be examined:

  • Priority setting
  • Project review and decision making
  • Program outreach and communications
  • Document/file review (e.g., TB submissions; logic model; RMAF; program and regional prospectuses and reports; meeting minutes; other program documents)
  • Interviews with NSC members; HSP Secretariat; RIB members; regional coordinators; funding recipients
  • Funding proponents/recipients survey
9. Is performance data collected against program activities/outcomes? If so, is collected information used to inform senior management/
decision makers?
  • Performance data are collected against program outputs and outcomes and are used to inform decision making
  • Presence/absence of populated performance data system with reliable and timely data
  • Evidence of decisions based on performance information
  • Extent to which performance measurement activities vary between regions
  • Document/file review (e.g., RMAF; planning documents; meeting minutes and decisions
  • Review of HSP Online Tracking System performance data)
  • Interviews with HSP Secretariat; RIB members; regional coordinators
  • Funding proponents/recipients survey
  • Case studies
10. What are the best practices and lessons learned from HSP?
  • Identified learnings and best practices
  • Identified learnings and best practices
  • Document review
  • Analyses completed for questions 1–9
  • Interviews with NSC members; HSP Secretariat; RIB members; regional coordinators; funding recipients
  • Case studies

Previous page | ToC | Next page