Evaluation Issues, Questions, Indicators, Data Sources and Data Collection Methods
Evaluation Questions |
Statement of
What Should Be Observed |
Indicators |
Information Sources/ Methods |
Relevance |
1. Is there a legitimate and necessary role for federal government in HSP? |
- HSP is aligned with identified SAR, departmental and federal priorities
- Program delivery complements but does not duplicate other programs
|
- Demonstration of a clear HSP mandate that is aligned with federal government jurisdiction
- Extent to which HSP’s goals and objectives correspond to Environment Canada/DFO/PCA strategic directions, federal government priorities and National Strategy for the Protection of Species at Risk
- Presence/absence of other programs that complement or duplicate the objectives and/or activities of HSP
|
- Document review (e.g., legislation/policies; Treasury Board submissions; speeches from the Throne and budgets; annual reports; departmental performance reports (DPRs) and reports on plans and priorities (RPPs))
- Interviews with NSC members; HSP Secretariat; RIB members; regional coordinators; experts/recovery team members; provincial/territorial representatives
|
2. Is HSP connected with environmental needs? |
- HSP serves the environmental needs of species at risk
|
- Demonstration that HSP addresses identified environmental needs regarding species at risk inhabiting non-federal lands and waters
|
- Document and literature review
- Interviews with NSC members; HSP Secretariat; RIB members; regional coordinators; experts/recovery team members; provincial representatives; funding recipients
- Funding proponents/recipients survey
- Case studies
|
Success/Impact |
3. To what extent have the intended immediate outcomes been achieved as a result of HSP? |
- Species-at-risk habitats protected
- Habitats for species at risk enhanced or restored
- Threats to species-at-risk individuals and populations reduced
- Canadians informed about species at risk and support conservation
|
- Extent to which species-at-risk habitats have been enhanced, protected or restored as a result of HSP (e.g., increase in the number of ha/km of shoreline protected or improved and number of participating landowners)
- Extent to which threats to species at risk have been reduced as a result of HSP (e.g., increase in the number of species-at-risk individuals protected)
- Evidence of HSP information activities reaching target groups of Canadians (e.g., increase in the number of people reached and participants engaged)
|
- Document review (e.g. , project final reports, program annual reports, other program documents)
- Review of HSP Online Tracking System for performance data
- Interviews with NSC members; HSP Secretariat; RIB members; regional coordinators; experts/recovery team members; provincial representatives; funding recipients
- Funding proponents/recipients survey
- Case studies
|
4. To what extent have the intended intermediate outcomes been achieved as a result of HSP? |
- Populations of species at risk are increased
- Species listed as at risk are delisted
- Total number of species listed as at risk is reduced
- Canadians are engaged in species-at-risk conservation
|
- Percentage change of select species-at-risk populations attributable to HSP activities
- Number of species listed as at risk targeted by HSP that have been delisted
- Percentage change in the total number of listed species at risk targeted by HSP
- Change in the level of stakeholders engagement in stewardship activities as a result of HSP (e.g., actions identified in recovery strategies)
- Opinions of stakeholders and tangible examples of achievement of intended intermediate outcomes
|
- Document review
- Review of HSP Online Tracking System for performance data
- Interviews with NSC members; HSP Secretariat; RIB members; regional coordinators; experts/recovery team members; provincial representatives; funding recipients
- Funding proponents/recipients survey
- Case studies
|
5. Have there been any unintended (positive or negative) outcomes that can be attributed to HSP? If so, were any actions taken as a result of these? |
- Unintended outcomes are present that can be attributed to the program
- Where appropriate actions to address unintended outcomes were taken
|
- Presence/absence of unintended outcomes
- Where appropriate, documented management actions and/or lessons learned from unintended outcomes
|
- Document/file review (e.g., meeting minutes and presentations; project and annual reports)
- Interviews with Secretariat; RIB members; regional coordinators; experts/recovery team members; provincial representatives; funding recipients
- Funding proponents/recipients survey
|
6. Are there any external factors outside of HSP that influence the success of the program? |
- Factors external to HSP are identified that have influenced the achievement of results
|
- Evidence of factors outside HSP that have influenced the achievement of intended outcomes (e.g., partner capacity and cooperation, information about species at risk, other policies and incentives)
- Where appropriate, documented management actions to address the influence of external factors
|
- Document/file review (e.g., planning documents; meeting minutes; project and annual reports; other program documents)
- Interviews with NSC members; HSP Secretariat; RIB members; regional coordinators; experts/recovery team members; provincial representatives; funding recipients
|
Cost-Effectiveness |
7. Are the most appropriate and efficient means being used to achieve HSP objectives? How could the efficiency of HSP activities be improved? |
- Alternative design/ delivery methods that are less costly/effort-intensive, enable quicker achievement of results, and /or enable achievement of better results
- Evidence of leveraging of funds from non-federal sources
- HSP activities show opportunities for efficiency increases
|
- Comparison of HSP activities to other similar programs
- Resources leveraged from contributions and their associated impact
- Opinion of key informants on the ability of program elements to achieve intended results, compared to alternative design/delivery approaches
- Opinions of key informants on whether HSP investments are a good use of public funds and whether the cost of producing outputs is as low as possible
- Opinions of key informants on how the efficiency of HSP activities could be improved
- Cost analysis
|
- Document review (e.g., program design reports; available documents on other programs)
- Review of program/project financial data
- Interviews with NSC members; HSP Secretariat; RIB members; regional coordinators; experts/recovery team members; provincial representatives; funding recipients
- Case studies
- Funding proponents/recipients survey
|
Design and Delivery |
8. Are program activities, processes and governance structures adequate for achieving expected HSP results? |
- HSP design is aligned with its overall mandate, identified needs and intended program outcomes
- Program processes are adequate and effective
- Roles, responsibilities and accountability are clear, adequate and duly implemented
|
- Soundness of logical linkages between program mandate, activities, outputs and intended outcomes
- Defined program processes, roles, responsibilities and accountability
- Opinions of key informants on the adequacy and effectiveness of program activities, processes and governance structures
Specific processes to be examined:
- Priority setting
- Project review and decision making
- Program outreach and communications
|
- Document/file review (e.g., TB submissions; logic model; RMAF; program and regional prospectuses and reports; meeting minutes; other program documents)
- Interviews with NSC members; HSP Secretariat; RIB members; regional coordinators; funding recipients
- Funding proponents/recipients survey
|
9. Is performance data collected against program activities/outcomes? If so, is collected information used to inform senior management/
decision makers? |
- Performance data are collected against program outputs and outcomes and are used to inform decision making
|
- Presence/absence of populated performance data system with reliable and timely data
- Evidence of decisions based on performance information
- Extent to which performance measurement activities vary between regions
|
- Document/file review (e.g., RMAF; planning documents; meeting minutes and decisions
- Review of HSP Online Tracking System performance data)
- Interviews with HSP Secretariat; RIB members; regional coordinators
- Funding proponents/recipients survey
- Case studies
|
10. What are the best practices and lessons learned from HSP? |
- Identified learnings and best practices
|
- Identified learnings and best practices
|
- Document review
- Analyses completed for questions 1–9
- Interviews with NSC members; HSP Secretariat; RIB members; regional coordinators; funding recipients
- Case studies
|