Evaluation of the Security and Prosperity Partnership of North America (SPP) Pilot Project on Reducing Emissions from Vehicles and Engines (PPRE)

July 2011

Previous page | Table of Contents | Next page

Executive Summary

Scope and Objectives of the Evaluation

A formal, independent evaluation is required to assess the relevance and performance of the Pilot Project on Reducing Emissions from Vehicles and Engines (PPRE). The scope of the evaluation is to cover the three years of the program and all related project activities. This requirement fulfills the commitment made in the original program design for the PPRE. As per the requirements presented in the Directive on the Evaluation Function, this evaluation will address value for money by including clear and valid conclusions about the relevance and performance of the pilot program. As the funding is not ongoing, and there is no decision required on program renewal, recommendations are not part of this evaluation. Rather, the focus is on lessons learned that may inform related future work and regulatory development.

Pilot Project on Reducing Emissions from Vehicles and Engines

The PPRE was an Environment Canada initiative, under the Security and Prosperity Partnership of North America (SPP), funded at the level of $1.2 million over three years (2008-11). The objectives of this pilot project were for Environment Canada and the United States Environmental Protection Agency (U.S. EPA) to collaborate on initiatives aimed at reducing emissions from vehicles and engines in order to better utilize the capabilities of each agency, and to share information on innovative programs such as retrofitting diesel engines for lower emissions. Funding was required to work on collaborative initiatives with the U.S. EPA, and to work with stakeholders to reduce emissions from diesel buses and heavy-duty trucks that are currently on the road.

Top of Page

Evaluation Methodology and Design

The evaluation addressed five key issues: (1) Continued need for the program; (2) Alignment with government priorities; (3) Alignment with federal roles and responsibilities; (4) Achievement of expected outcomes; and (5) Demonstrated efficiency and economy. To examine these issues, and the specific evaluation questions related to each issue, the evaluation employed three lines of inquiry: (1) document review; (2) literature review and comparison of similar programs; and (3) key informant interviews. The evaluation scope focused on Environment Canada’s pilot project to work with stakeholders (within Canada) to reduce emissions from diesel buses and heavy-duty trucks that are currently on the road.

Top of Page

Evaluation Conclusions

The key findings from the evaluation, detailed in the report, are summarized below.

Relevance

The pilot project and its respective activities are relevant and continue to be needed to address heavy-duty diesel (HDD) greenhouse gas emissions and criteria air contaminants produced by public fleets.
The pilot project is directly aligned with federal government priorities and the strategic outcomes of Environment Canada to take action on climate change and reduce air emissions.
The activities of the pilot program are aligned with federal roles and responsibilities.

Performance – Project Design and Delivery

Performance information is provided by funded partners, primarily at the output level, in a timely and relevant manner. Data pertaining to the project’s performance indicator are currently being collected and will be reviewed at the end of the pilot. The evaluation did not find evidence of a formal internal reporting mechanism to senior managers or federal partners, though information was shared throughout the project on an informal basis.
The project was designed appropriately and included a clear and plausible link between program activities, outputs and expected results. The resources allocated to this project were commensurate with the expected results.
The pilot project identified that access to funding and the ability to work with the variety of players who are involved with municipal fleets (e.g. ranging from municipal governments to subcontractors and third parties) may be barriers to implementing a similar project on a national scale.
On behalf of Environment Canada, project partners engaged municipal fleet operators regularly and communicated the outputs from project activities in a timely and relevant manner. Correspondence and communication within the federal community was informal and in some cases inconsistent.
The pilot project was successful in developing a best practices guide and a resources guide for municipal fleet managers, and in providing advice on how to move forward with addressing emissions from school buses.

Performance – Effectiveness

The pilot project successfully generated a number of important outputs, including a best practices guide and a resources guide, fact sheets, and multiple reports that increased the level of knowledge about and number of approaches to addressing the on-road HDD municipal fleet and school bus sectors. One planned output (a baseline and profile of municipal fleets in Canada) was not achieved. Data are not yet available to assess the achievement of the project’s expected outcome.
The evaluation did not find any negative unintended outcomes.
Environment Canada successfully partnered with organizations that were experts in the subject matter and/or maintained a comprehensive understanding of the key organizations and individuals within the municipal fleet sector. In one case, the Department successfully leveraged funding from an organization to support the project.
The evaluation found that the project is on track to achieving its intended outcome in an economical manner.

Top of Page

Lessons Learned

The evaluation identified a number of lessons learned, and suggestions on how to improve the delivery of similar projects complementary to regulatory development. These are detailed in Section 6, and are summarized below.

Key Lessons Learned from the PPRE:

Allocating an appropriate amount of time to designing a program/project can lead to relevant and valuable results. Environment Canada allocated one year to design a successful pilot project that was relevant and met a demonstrable need that was not duplicative of other federal efforts.
Partnering with organizations that maintain expertise and existing networks can lead to effective results. Environment Canada worked with non-governmental organizations (NGOs) that maintained existing networks and a comprehensive understanding of municipal fleets across Canada. This approach enhanced program efficiencies by creating immediate access to existing networks, and provided greater assurance that project deliverables were credible and were useful to the municipal fleet community.
When provided with the appropriate tools and funding, Canadian municipalities can engage in actions to reduce emissions from their fleet vehicles. The PPRE has shown that by building the capacity of municipalities, providing tools, and providing assistance in accessing funds, emission reductions can be supported and achieved.

Suggestions for Future Projects:

A one-window approach, providing information and resources to reduce emissions from all types of vehicles in the municipal fleet, would be beneficial. There is merit in addressing the emissions from all vehicle types that are present in the municipal fleet more broadly, rather than just focusing on HDD. Fleet managers deal with multiple types of vehicles and make integrated decisions that can best be informed and supported by a more comprehensive approach to addressing emissions from all vehicle types.
Long-term funding is required to ensure action is taken to reduce fleet emissions. The pilot project has increased the general level of awareness of municipal fleet managers and has provided a number of resources to reduce the emissions associated with municipal fleets. However, the pilot program has also underlined a need for long-term funding to implement solutions to reduce emissions produced by municipal fleets. This pilot project has demonstrated that long-term funding from sources with municipal jurisdiction should be identified.
The Government of Canada can play an important role in identifying innovative technologies for reducing vehicle emissions. The Government of Canada can play an important role in testing of technologies for reducing on-road vehicle emissions. The PPRE-funded project in Ontario led to the identification of retrofit technology that has a proven ability to reduce emissions produced by school buses.
Environment Canada is best placed as an enabler for reducing vehicle emissions. The evaluation highlighted that this pilot project was a unique opportunity for Environment Canada to lead a pilot project that identified a gap, was complementary to HDD regulations, and addressed greenhouse gases and air pollutants. However, other federal departments, such as Natural Resources Canada, are now better placed to lead comprehensive capacity-building programs in this area, because of their mandate and emerging work under the FleetSmart program.
A national emissions baseline for municipal fleets is required. The pilot project identified the need for a national emissions baseline for municipal fleets, but found it challenging to complete. In order to target cost-effective emission reduction activities and to measure change with municipal fleets, a baseline is required.
The impact of a capacity-building / outreach program is maximized when delivered jointly with a suite of like-minded programs, including regulations/standards. Although education and capacity building are important tools to help municipalities take action to reduce their emissions, a suite of complementary and reinforcing initiatives, including regulations, provides them with the push and pull required for taking substantive action within their jurisdictions.

Previous page | Table of Contents | Next page