Evaluation of the National Vehicle Scrappage Program

Final Report

July 12, 2011

Previous page | Table of Contents | Next page

Annex 3 Evaluation Issues and Questions

Relevance

Was the Vehicle Scrappage Program consistent with, and did it contribute to, the federal government priorities, and did it address actual needs?

Question 1. a)

Did the Vehicle Scrappage Program respond to an identified need? What was that need?

Indicators

Demonstration of societal/environmental need
Views on connection of program objectives with societal/environmental need

Data Sources/Methods

Document review
KI interviews
Case studies (recycling code of practice)

TB Policy Issue Addressed

Issue #1: Continued Need for Program

Top of Page

Question 1. b)

Is there a continued need for the program?

Indicators

Degree to which gaps would exist in addressing societal/environmental need in absence of the program
Views on whether there is a continued need for the Vehicle Scrappage Program

Data Sources/Methods

Document review
Literature review (including statistics on age of vehicles in use)
KI interviews
Case studies (recycling code of practice)

TB Policy Issue Addressed

Issue #1: Continued Need for Program

Top of Page

Question 2.

Did the program complement or duplicate other related federal and/or provincial government programs (i.e., Transport Canada programs, provincial emission testing programs, etc.)?

Indicators

Presence/absence of other government programs (federal and provincial) that complement or duplicate the objectives of the program
Views on whether the program complemented or duplicated other related programs

Data Sources/Methods

Document review
Literature review
KI interviews

TB Policy Issue Addressed

Issue #1: Continued Need for Program

Top of Page

Question 3.

Was the Vehicle Scrappage Program aligned with departmental and federal government priorities?

Indicators

Extent to which the program’s objectives corresponded to recent/current federal government priorities (e.g., CAA objectives)
Extent to which the program’s objectives aligned with recent/current departmental strategic outcomes
Views on the alignment of the program’s objectives with recent/current departmental and federal government priorities

Data Sources/Methods

Document review
Literature review
KI interviews

TB Policy Issue Addressed

Issue #2: Alignment with Federal Government Priorities

Top of Page

Question 4.

Was the Vehicle Scrappage Program consistent with the Department’s mandate and with federal roles and responsibilities?

Indicators

Extent to which program mandate was aligned with departmental mandate and federal government jurisdiction
Views on the appropriateness of departmental and federal involvement

Data Sources/Methods

Document review
Literature review
KI interviews

TB Policy Issue Addressed

Issue #3: Alignment with Federal Roles and Responsibilities

Top of Page

Performance

Has the Vehicle Scrappage Program achieved its intended outcomes? Were the most appropriate, efficient and economic means used to achieve outcomes?

Top of Page

Question 5.

Were the objectives of the program realistic and adequately defined?

Indicators

Demonstration that the Vehicle Scrappage Program had realistic objectives
Degree to which the program’s objectives are clearly understood by key stakeholders
Views on the viability and clarity of program objectives

Data Sources/Methods

Document review
Literature review
KI interviews

TB Policy Issue Addressed

Issue #4: Achievement of Expected Outcomes [Design and Delivery]

Top of Page

Question 6. a)

Was the design of the Vehicle Scrappage Program appropriate for achieving expected program results?

Indicators

Plausibility of link between program activities, outputs and intended outcomes
Extent to which program processes and governance structure, including roles, responsibilities and accountabilities, were clearly defined and understood
Views on appropriateness of program activities, processes and governance structure

Data Sources/Methods

Document review
Literature review
KI interviews
Case studies

TB Policy Issue Addressed

Issue #4: Achievement of Expected Outcomes [Design and Delivery]

Top of Page

Question 6. b)

Was the funding of the program commensurate with its objectives and the needs identified?

Indicators

Evidence of the degree to which program resources and capacity were commensurate with expected program results
Views on adequacy of program funding

Data Sources/Methods

Document review
Literature review
KI interviews

TB Policy Issue Addressed

Issue #4: Achievement of Expected Outcomes [Design and Delivery]

Top of Page

Question 7. a)

Were participants satisfied with the administration of the program by both the Department and the not-for-profit organizations?

Indicators

Level of participants’ satisfaction with administration of the program by the Department and the NFPs

Data Sources/Methods

Secondary data analysis (tracking research on program participants)

TB Policy Issue Addressed

Issue #4: Achievement of Expected Outcomes [Design and Delivery]

Top of Page

Question 7. b)

Were participants satisfied with the incentives provided to reward them for scrapping their vehicle (type and value)?

Indicators

Level of satisfaction of participants with the incentives provided through the program

Data Sources/Methods

Secondary data analysis (tracking research on program participants)
Case studies

TB Policy Issue Addressed

Issue #4: Achievement of Expected Outcomes [Design and Delivery]

Top of Page

Question 8.

Were the achievements of the Vehicle Scrappage Program effectively and efficiently captured and disseminated to appropriate audiences (i.e., key stakeholders and the general public)?

Indicators

Degree to which program activities and results were effectively communicated to key stakeholders (including provision of input into CAA Horizontal Management Accountability and Reporting Framework)
Availability of information on program results for the Canadian public
Views on extent to which program achievements were effectively and efficiently captured and shared with key stakeholders and general public

Data Sources/Methods

Document review
Literature review
KI interviews
Case studies

TB Policy Issue Addressed

Issue #4: Achievement of Expected Outcomes [Design and Delivery]

Top of Page

Question 9.

Have appropriate performance data been collected, captured and safeguarded? If so, has this information been used to inform senior management / decision makers?

Indicators

Existence of an effective performance measurement strategy
Extent to which performance data are reliable, timely and relevant
Extent to which performance data informed/supported EC’s decision-making processes
Views on strengths and weaknesses of the Vehicle Scrappage Performance Monitoring Framework

Data Sources/Methods

Document review
Secondary data analysis (program performance data)
KI interviews

TB Policy Issue Addressed

Issue #4: Achievement of Expected Outcomes [Design and Delivery]

Top of Page

Question 10. a)

What are the lessons learned (both strengths and weaknesses) from the program?

Indicators

Views about the program’s strengths, best practices and weaknesses (what worked well, what has not worked well, and what could have been done better)

Data Sources/Methods

Document review
KI interviews
Case studies

TB Policy Issue Addressed

Issue #4: Achievement of Expected Outcomes [Design and Delivery]

Top of Page

Question 10. b)

Are there follow-up activities or programs that could build on the success achieved by this program?

Indicators

Extent to which program activities could lead to further future opportunities to meet the overall CAA objectives or other environmental priorities

Data Sources/Methods

Document review
Literature review
KI interviews
Case studies

TB Policy Issue Addressed

Issue #4: Achievement of Expected Outcomes [Design and Delivery]

Top of Page

Question 11.

To what extent have the intended outcomes been achieved as a result of the Vehicle Scrappage Program?

Question 11. a) Short-term outcomes

Canadians in all provinces have access to incentives to reward them for removing older vehicles from the road

Indicators

Number of provinces and territories (agreed upon in approved operating plan) in which program was available
Number of provinces and territories (agreed upon in approved operating plan) in which trained recyclers were available
Views on extent to which outcome has been achieved

Data Sources/Methods

Secondary data analysis (program database)
Document review
KI interviews
Case studies

TB Policy Issue Addressed

Issue #4:Achievement of Expected Outcomes [Effectiveness]

Top of Page

Question 11. b) Short-term outcomes

Increased awareness by the public of the program and the environmental impacts of older vehicles

Indicators

Change in level of awareness among Canadians (the general population and owners of older vehicles) about the environmental impacts of older vehicles
Change in attitude toward the use of older vehicles
Change in level of awareness among Canadians about the program
Degree to which the public believed the program was a good idea
Number of requests for information about incentives received via the website and/or 1-800 O-Canada (Service Canada)
Views on extent to which outcome has been achieved

Data Sources/Methods

Secondary data analysis (tracking research on public)
Document review
KI interviews

TB Policy Issue Addressed

Issue #4:Achievement of Expected Outcomes [Effectiveness]

Top of Page

Question 11. c) Short-term outcomes

Increased awareness by recyclers of environmentally safe recycling practices

Indicators

Recyclers’ level of acceptance of and attitude toward the recycling code of practice
Views on extent to which outcome has been achieved

Data Sources/Methods

Secondary data analysis (Vehicle Recyclers Survey)
KI interviews
Document review
Case studies

TB Policy Issue Addressed

Issue #4:Achievement of Expected Outcomes [Effectiveness]

Top of Page

Question 11. d) Medium-term outcomes

Fewer older, high-polluting personal vehicles on the road

Indicators

Number of older high-polluting personal vehicles scrapped as part of the program
Reported influence of the program on participants’ decision to recycle their vehicle
Views on extent to which outcome has been achieved

Data Sources/Methods

Secondary data analysis (program database and tracking research on program participants)
Document review
KI interviews
Case studies

TB Policy Issue Addressed

Issue #4:Achievement of Expected Outcomes [Effectiveness]

Top of Page

Question 11. e) Medium-term outcomes

More sustainable transportation modes being used for personal transportation

Indicators

Uptake of incentives that do not involve the ownership of another vehicle
Use of sustainable transportation during the incentive life (e.g., one year of free transit passes)
Potential use of sustainable transportation beyond incentive life
Views on extent to which outcome has been achieved

Data Sources/Methods

Secondary data analysis (program database and tracking research on program participants)
Document review
KI interviews
Case studies

TB Policy Issue Addressed

Issue #4:Achievement of Expected Outcomes [Effectiveness]

Top of Page

Question 11. f) Medium-term outcomes

Environmentally safe practices in use for recycling of vehicles scrapped through the program

Indicators

Level of adherence with code of practice by vehicle recyclers
Level of support from provincial recycling associations and provincial governments for code of practice
Extent of adoption of code of practice by provinces and other stakeholders
Views on extent to which outcome has been achieved

Data Sources/Methods

KI interviews
Document review
Literature review
Case studies

TB Policy Issue Addressed

Issue #4:Achievement of Expected Outcomes [Effectiveness]

Top of Page

Question 11. g) Long-term outcomes

Air pollutants are reduced

Indicators

Calculated reductions in smog-forming emissions
Views on extent to which outcome has been achieved

Data Sources/Methods

Secondary data analysis (program database)
Document review
Literature review
KI interviews

TB Policy Issue Addressed

Issue #4:Achievement of Expected Outcomes [Effectiveness]

Top of Page

Question 11. h) Long-term outcomes

GHG emissions are reduced

Indicators

Calculated reductions in GHG emissions
Views on extent to which outcome has been achieved

Data Sources/Methods

Secondary data analysis (program database)
Document review
Literature review
KI interviews

TB Policy Issue Addressed

Issue #4:Achievement of Expected Outcomes [Effectiveness]

Top of Page

Question 11. i) Long-term outcomes

Release of harmful and toxic substances into the environment is prevented

Indicators

Number of vehicles recycled according to code of practice
Views on extent to which outcome has been achieved

Data Sources/Methods

Secondary data analysis (program database)
Document review
Literature review
KI interviews

TB Policy Issue Addressed

Issue #4:Achievement of Expected Outcomes [Effectiveness]

Top of Page

Question 12.

Did this program lead people to make use of other related federal and provincial government programs or vice-versa?

Indicators

Level of involvement of program participants in the use of other related programs
Views on extent to which outcome has occurred

Data Sources/Methods

Secondary data analysis (tracking research on program participants)
KI interviews
Case studies

TB Policy Issue Addressed

Issue #4:Achievement of Expected Outcomes [Effectiveness]

Top of Page

Question 13.

Were there any external factors outside of the Vehicle Scrappage Program that influenced (positively or negatively) the success of the program?

Indicators

Evidence of factors outside the program which have influenced the achievement of intended outcomes
Views on whether there were any external factors that helped or hindered the program’s success

Data Sources/Methods

Document review
Literature review
KI interviews
Case studies

TB Policy Issue Addressed

Issue #4:Achievement of Expected Outcomes [Effectiveness]

Top of Page

Question 14.

Were there any unintended positive or negative outcomes of the Vehicle Scrappage Program?

Indicators

Presence or absence of unintended outcomes
Views on whether unintended outcomes occurred as a result of the program

Data Sources/Methods

Document review
KI interviews
Case studies

TB Policy Issue Addressed

Issue #4:Achievement of Expected Outcomes [Effectiveness]

Top of Page

Question 15.

Did the Vehicle Scrappage Program undertake its activities and deliver its products in the most efficient manner?
Was federal funding leveraged to the extent planned?
How could the efficiency of the program be improved? program’s activities be improved?
Are there alternative, more efficient ways of delivering the program?

Indicators

Analysis of actual program operational costs in relation to the production of outputs
Analysis of program administrative costs in relation to total program costs
Views on whether the cost of producing program outputs was as low as possible
Amount of financial commitments from incentive partners
Amount of financial commitments and other support from provinces and territories toward the program’s delivery
Number of partnerships achieved by national NFP with incentive partners that supported the program’s goals
Views on how the efficiency of program activities could be improved
Comparison of program activities and products delivered by other similar programs, including U.S. Cash for Clunkers
Views on whether there are alternative, more efficient ways of delivering program activities and outputs

Data Sources/Methods

Document review
Financial analysis
Literature review
KI interviews

TB Policy Issue Addressed

Issue #5: Demonstration of Efficiency and Economy [Efficiency]

Top of Page

Question 16.

Did the Vehicle Scrappage Program achieve its intended outcomes in the most economical manner?

Indicators

Extent to which the program’s intended outcomes have been achieved at the least possible program cost
Views on whether good value was obtained with respect to the use of public funds
Evidence of / views on whether there are alternative program models that would achieve the same expected outcomes at a lower cost

Data Sources/Methods

Document review
Financial analysis
Literature review
KI interviews

TB Policy Issue Addressed

Issue #5: Demonstration of Efficiency and Economy [Economy]

Previous page | Table of Contents | Next page