Evaluation of the EcoAction Community Funding Program

Evaluation Issue

Indicators

Data Sources

RELEVANCE: Does the program remain consistent with and contribute to the federal government priorities and address actual needs?

• Demonstration of mandate that is aligned with a public good
• Appropriate level of government is involved
• Brokers inter/intra-governmental relations (F/P/T and First Nations)
• Evidence of consistency with Government of Canada priorities

• Document / File Review
• KI Interviews
• Analysis of 2002 and 2008 Surveys
• Survey of Non-Funded Applicants

2. Is the program connected with societal /environmental needs? To what extent is EcoAction addressing those needs?

• Demonstration of the utility/rationale for the program
• Gaps would exist in coverage

• Document / File Review
• KI interviews
• Analysis of 2002 and 2008 Surveys
• Survey of Non-Funded Applicants
• Analysis of MIS Data

SUCCESS: Has the program achieved its intended outcomes?

3. To what extent is the EcoAction program reaching its intended target audiences? What factors have facilitated or impeded the program’s ability to reach its target audiences?

• Reach is analyzed and targeted, and both the reach and activities are connected to societal/environmental requirements

• Document / File Review
• KI interviews
• Analysis of 2002 and 2008 Surveys
• Survey of Non-Funded Applicants
• Analysis of MIS Data

4. To what extent has the program progressed towards its intended immediate outcomes as identified in its logic model? What are the barriers to success?

• Extent to which these outcomes would / would not have occurred without the program
• Extent to which there were barriers / factors which prevented the program from being successful in this regard

• Document / File Review
• KI interviews
• Analysis of 2002 and 2008 Surveys
• Survey of Non-Funded Applicants
• Analysis of MIS Data 

• Communities take action on the reduction of GHG and other local environment priority issues

• Number, size, geographic location of communities with funded project

• Increased capacity and skills of communities to act on priority environmental issues

• Self-assessment of communities against capacity building
• Total number of volunteer person-years
• % of respondents who agreed with statement, “My most recent EcoAction project has helped increase my organization’s capacity to deliver environmental programs in the community” (Q25e).

• Improved access to information for funding recipients and supporting sponsors

• Perceptions of funded applicants and partners on accessibility forms and tools

• Improved knowledge sharing between and among EcoAction, funding recipients and their communities

• % of respondents who agreed with statement, “My most recent EcoAction project has assisted in the sharing of information and knowledge within my community” (Q25d).

• Improved networking, creation of partnerships between communities and other sectors

• Value of in-kind support
• Value of leveraged funds
• Number and type of sponsors formally committed
• Recipients’ perception of partnerships (Type of organization and how it assisted with the project)

5. To what extent has the program progressed towards the achievement of the intermediate outcomes identified in its logic model? What were the barriers to success?

• Extent to which these outcomes are attributable to the program
• Extent to which there were barriers / factors which prevented the program from being successful in this regard

• Document / File Review
• KI interviews
• Analysis of 2002 and 2008 Surveys
• Survey of Non-Funded Applicants
• Analysis of MIS Data

• Improved knowledge and skills within communities and among individuals to preserve and protect the environment

• % of respondents who agreed with statement, “My most recent EcoAction project has helped to improve the knowledge and skills within my community to preserve and protect the environment” (Q25c).

• Improved community capacity, skills and knowledge to build and sustain projects into the future

• % of funded projects that continue after the funding period has ended and degree of continuation 
• Degree and nature of continued partnerships 
• Reasons for projects not continuing
• Soci-economic indicators of MIS

• Community groups achieve measurable results for the protection, rehabilitation and/or enhancement of natural environment

N/A: To be measured through other outcomes listed below (i.e., reduction in emissions that contribute to other air quality issues (e.g., smog), reduction in and diversion from the use of hazardous substances that affect water quality, and protection of wildlife and plants, and protection and improvement of their habitat)

• Reduction in emissions that contribute to climate change through increased awareness and participation of Canadians in the One-Tonne Challenge

N/A: the One-Tonne-Challenge was cancelled in 2007

• Reduction in emissions that contribute to other air quality issues (e.g., smog)

• Clean Air, Climate Change indicators in MIS
• Changes overtime

• Reduction in and diversion from the use of hazardous substances that affect water quality

• Clean water indicators in MIS
• Changes overtime

• Protection of wildlife and plants, and protection and improvement of their habitat.

• Nature indicators in MIS
• Changes overtime

6. To what extent has the program contributed to its stated ultimate outcomes? What were the barriers to success?

• Extent to which EcoAction has contributed to these
• Extent to which there were barriers / factors which prevented the program from being successful in this regard

• Document / File Review
• KI interviews
• Analysis of 2002 and 2008 Surveys
• Survey of Non-Funded Applicants
• Analysis of MIS Data 

• Healthier and sustainable communities are created

• % of respondents who agreed with statement from 2008 Client Survey: 
  • “My most recent EcoAction project has helped create a healthier community” (Q25a); and 
  • “My most recent EcoAction project has helped create a more sustainable community” (Q25b).

• Individual Canadians make more responsible environmental decisions

• % of respondents who agreed with statement from 2008 Client Survey:
  • “My most recent EcoAction project has helped raise community awareness on local environmental issues” (Q25f); and  
  • “My most recent EcoAction project has helped educate the community on environmental issues” (Q25g).

Cost Effectiveness: Are the most appropriate, cost effective and efficient means being used to achieve outcomes?

7. Has the program provided value for federal dollars spent?

• Analysis of costs and impacts of program compared with achievement of program outcomes
• O&M as percent of G&C
• Changes overtime, for example:
  • Number of FTEs
  • Number of projects approved
• % of EcoAction funding over total project value

• Document / File Review
• KI Interviews
• Analysis of 2002 and 2008 Surveys
• Analysis of MIS Data

8. Are others involved in the same areas of activities and/or share similar objectives? How is duplication avoided and complementarity achieved?

• Analysis of comparable programs as to duplication and complementarity 
• Extent to which those participating in the program are aware / have used other government program which may duplicate, overlap or complement with EcoAction

• Document /File Review
• KI Interviews

9. Are there any alternative, more cost-effective ways of achieving the stated outcomes?

• Extent to which the program budget is appropriate in consideration of the stated objectives (overall, by region, by priority, etc.)
• Identified suggestions for improvement which would make the program more cost effective
• Lessons learned from regions in terms of cost effectiveness
• Departmental financial records for financial resources of program (total, by region, by component)
• Number of proposals received by FTE or O&M (total, by region)

• Document / File Review
• KI Interviews
• Analysis of MIS Data

Design and Delivery: Is the program designed and delivered in the best possible way?

10. To what extent are various target groups satisfied with the program and the changes to the program?

• Satisfaction of various target groups

• Document / File Review
• KI Interviews
• Analysis of 2002 and 2008 Surveys
• Survey of Non-Funded Applicants

11. How effective is the governance structure?

• Extent to which the governance structure has worked effectively
• Identified suggestions for improvement to the governance structure
• Extent to which the program demonstrated principles of good governance (i.e., participatory, transparent, responsive, consensus-oriented, equitable and inclusive, effective and efficient, accountable)

• Document / File Review
• KI Interview

12. Is the performance data collected against program activities/outcomes? If so, is collected information used to inform senior management/decision makers?

• Presence of populated performance date system
• Decisions based on performance information

• Document / File Review
• KI Interviews

13. What are the best practices and lessons learned from this program?

• Identified lessons learned and best practices
• Identified strengths and weaknesses
• Factors that contribute to/detract from the achievement of results

• KI Interviews
• Analysis of 2002 and 2008