Evaluation of the Security and Prosperity Partnership of North America (SPP) Pilot Project on Reducing Emissions from Vehicles and Engines (PPRE)

July 2011

Previous page | Table of Contents | Next page

4 Findings by Evaluation Issue

This section outlines evaluation findings for each of the defined evaluation issues and questions. Findings and ratings are presented by evaluation issue for the PPRE, in keeping with the protocol of Environment Canada’s Evaluation Division, with specific comments included under each evaluation question.

4.1 Rating of Findings

Ratings have been provided to indicate the degree to which the PPRE has addressed the key evaluation criteria. The rating is assessed according to the following chart provided by Environment Canada’s Evaluation Division. A summary table of these ratings/findings is provided in Annex 4.

Table 6: Evaluation Ratings Legend

Rating

Significance

Achieved

The intended outcomes or goals have been achieved or met

Progress Made; Attention Needed

Considerable progress has been made to meet the intended outcomes or goals, but attention is still needed

Little Progress; Priority for Attention

Little progress has been made to meet the intended outcomes or goals and attention is needed on a priority basis

N/A

A rating is not applicable

~

Outcomes achievement ratings are based solely on subjective evidence

Top of Page

4.2 Relevance – Continued Need for the Program

Evaluation Issues (and related questions)

Indicators

Overall Rating

1. Was there a need for the project? Is there a need for the project to continue?

  • Demonstration of environmental need
  • Demonstration that the pilot complements and does not duplicate other similar programs (CARA, ecoFREIGHT, etc.)
  • Demonstration that pilot addresses gaps in environmental need
  • Project activities and reach are connected to environmental need
  • Views of stakeholders and project managers on the connection of project objectives with environmental need

Achieved

Summary

The evaluation found that there was a clear need for the pilot and a continued need to reduce emissions from municipal fleets. These vehicles are a significant contributor to local air pollution and GHG emissions, and pose a health risk to vulnerable populations (e.g., school children). The evaluation found that there are regional programs that target this need, but they are not national in scope, do not focus exclusively on on-road HDD vehicles, and remedial actions are not widespread. There are also similar federal programs, but the pilot was designed to complement these initiatives and address a specific area (i.e., on-road HDD including GHG and CAC emissions). The PPRE was designed to target an identified gap in meeting the needs of municipal fleet operators nationally. Due to the noted need, the overall rating has been assessed as Achieved.

Municipalities operate HDD vehicles as a normal part of their operations, for services such as the transport of school children, garbage collection and snow removal. However, municipal fleets are a key contributor to the environmental and health impacts of municipal operations. Burning diesel fuel creates GHG emissions that contribute to climate change, and CACs that contribute to smog and acid rain. On average, Canadian municipal fleets are responsible for three to five percent of a municipality’s total GHG emissions, and consume billions of litres of fuel each year. In addition, HDD vehicles such as school buses can expose on-board children to elevated levels of air pollutants (see sidebar), adding to their environmental exposure. In particular regions (e.g., British Columbia and southern Ontario), local air quality is of particular concern and international agreements have been established to address air quality (e.g., Canada–U.S. Air Quality Agreement).

Although federal and provincial programs address emissions from HDD vehicles (see Annex 2), the PPRE was designed to complement these programs and target on-road legacy public vehicles that will continue to emit GHGs and CACs for many years. The pilot was designed not just to promote technology and tools to reduce emissions, but also to address needs specific to municipalities with respect to obtaining support for action (e.g., demonstrated economic benefits, business cases for action, etc.).

As part of the PPRE program design, Environment Canada consulted with federal partners and examined current initiatives to ensure the pilot project filled an identified gap. Partners indicated that “municipalities are committed to reducing emissions from their fleets, but are unfamiliar with the appropriate technological and behavioural adaptations.” Furthermore, the literature review completed for this evaluation noted that regional programs in British Columbia, Ontario and Nova Scotia work with industry and/or fleet managers to reduce emissions; however, these programs are not national in scope. Table 7 summarizes these regional capacity-building programs and compares them to the PPRE (see Annex 2 for details).

Table 7: Comparison of Capacity-Building Programs

Fleet Challenge Ontario (Municipal Fleet Review)

Funding source

Provincial government

Delivery partner

One NGO

Target audience

Municipalities

Types of vehicles

On-road vehicles
Off-road vehicles (vehicle selection)

Objective

To support green management decisions and demonstrate ways to improve operations, thereby reducing costs and GHG emissions

Mechanisms/activities

Fleet audits

Workshops for fleet managers

Networking/forums for fleet managers

Toolkits and guides

Links to other resources on website

Testing of reduction technologies

Duration

2008 (pilot project)
2009 to 2012

 


BC Air Action Plan (Action #10 only)

Funding source

Provincial government

Delivery partner

One NGO

Target audience

Municipalities
Commercial sector

Types of vehicles

On-road vehicles

Objective

To help owners of commercial and public sector vehicle fleets improve their fuel efficiency and reduce emissions

Mechanisms/activities

Fleet audits

Workshops for fleet managers

Networking/forums for fleet managers

Toolkits and guides

Links to other resources on website

Testing of reduction technologies

Duration

2007 to 2011

 


Clean Nova Scotia - Fleetwiser Program

Funding source

Federal government

Delivery partner

One NGO

Target audience

Municipalities
Utilities (energy)

Types of vehicles

On-road vehicles

Objective

To create a cleaner, healthier environment by informing, enabling and inspiring Nova Scotian fleet operators to respect and consider the environment in all their choices

Mechanisms/activities

Fleet audits

Workshops for fleet managers

Networking/forums for fleet managers

Toolkits and guides

Links to other resources on website

Testing of reduction technologies

Duration

2009 to 2011

 


PPRE Pilot Project

Funding source

Federal government

Delivery partner

Three NGOs
One consulting firm

Target audience

Municipalities
School bus operators and districts

Types of vehicles

On-road HDD vehicles
Heavy-duty diesel engines

Objective

To reduce emissions (GHGs and CACs) from Canada’s on-road HDD municipal fleet by increasing knowledge and awareness and supporting tools for municipalities

Mechanisms/activities

Fleet audits

Workshops for fleet managers

Networking/forums for fleet managers

Toolkits and guides

Links to other resources on website

Testing of reduction technologies

Duration

2008 to 2011

Source

Evaluation Work Plan, February 2011
http://fmv.fcm.ca/Enviro-Fleet/

 

Top of Page

4.3 Relevance – Alignment with Federal Government Priorities

Evaluation Issues (and related questions)

Indicators

Overall Rating

2. Is the project aligned with federal government priorities?

  • Project’s objectives correspond to recent/current federal government priorities
  • Project’s objectives are aligned with current departmental strategic outcomes
  • Evidence of alignment with SPP, CARA, ecoFREIGHT and FleetSmart

Achieved

Summary

The PPRE’s objectives are clearly aligned with current federal government priorities, and with Environment Canada’s strategic outcomes to take action on climate change and reduce air emissions. The PPRE was found to be complementary to other federal programs such as CARA and ecoFREIGHT. However, it was found to be similar to NRCan’s FleetSmart program, although FleetSmart focuses broadly on GHG emissions and public and private vehicles, while the PPRE focused on both GHGs and CACs specifically for on-road public municipal fleets and school buses. Due to the alignment with federal priorities and outcomes, the overall rating has been assessed as Achieved.

When designed, the PPRE was aligned with the Government of Canada priority, identified in the 2008 Speech from the Throne, of “tackling climate change and preserving Canada’s environment by reducing greenhouse gas emissions 20 percent by 2020.”5 This commitment to addressing climate change is maintained in the 2010 Speech from the Throne.

The PPRE is aligned with Environment Canada’s priorities and strategic outcomes, as evidenced in the following Reports on Plans and Priorities (RPPs):

The PPRE complements the Government of Canada’s CARA by addressing emissions from existing vehicles and engines that could remain on Canada’s roads for up to 20 years. CARA focuses on the On-Road Vehicle and Engine Emission Regulations to reduce emissions from new HDD engines (starting with the 2007 model year). However, diesel vehicles built before 2007 will continue to emit comparatively high levels of nitrogen oxides (NOx) and PM over the next 20 years. The PPRE targets the on-road legacy public HDD vehicle fleet, such as municipal fleet vehicles, and explores how best to voluntarily reduce their emissions. Reducing emissions from the in-use public HDD fleet addresses two important Government of Canada priorities: reducing emissions (GHGs and CACs) and improving the health of Canadians.

The PPRE was designed to complement other federal programs such as Transport Canada’s ecoFREIGHT and NRCan’s FleetSmart (details of these programs are provided in Annex 2). It clearly complements ecoFREIGHT, which works with the freight transportation industry to increase the uptake of technologies and practices that reduce fuel consumption and emissions of CACs and GHGs. However, there is some overlap with the FleetSmart program, which offers free practical advice on how energy-efficient vehicles and business practices can reduce commercial and municipal fleet operating costs, improve productivity and increase competitiveness. NRCan’s transportation programs, in keeping with their PAA, indicate that they “aim to reduce GHG emissions from on-road transportation by encouraging drivers and fleet managers to use energy-efficient purchasing, driving and vehicle maintenance behaviours.” This is similar to the Environment Canada strategic outcomes. However, the focus of the FleetSmart program is GHGs (not air pollutants as well), and more on training and retrofits to promote energy efficiency specifically related to GHG emissions, while the PPRE focused particularly on capacity building and tools specific to HDD public municipal fleets and addressed both GHG and CAC emissions. Also, the PPRE focuses solely on public fleets (municipal fleets and school buses), while the FleetSmart program focuses on public and private vehicles.

Top of Page

4.4 Relevance – Consistency with Federal Roles

Evaluation Issues (and related questions)

Indicators

Overall Rating

3. Is the project consistent with federal roles and responsibilities?

  • Project mandate aligned with federal government jurisdiction
  • Views on the appropriateness of federal involvement

Achieved

Summary

Managing emissions is a partnership between multiple government jurisdictions and stakeholders. There is a federal role and responsibility for the Government of Canada to regulate emissions from vehicles and engines, and thus engaging in pilots such as the PPRE to complement such regulations is appropriate providing that federal authorities focus their efforts on facilitating and enabling such programs and projects with relevant partners. Responsibilities for building capacity with and reducing emissions from the in-use municipal fleet are also strongly aligned with provincial and municipal roles and responsibilities. Stakeholders viewed the federal role in this pilot as appropriate. Therefore, the overall rating has been assessed as ‘achieved’.

All levels of government have roles to play in addressing emissions from HDD vehicles. The environment is a matter of shared jurisdiction between the provincial and federal governments. Both levels of government derive their jurisdiction from distinct powers granted by the Constitution Act 1867. Both levels of government may have the authority to regulate the same emissions, albeit based upon different Constitutional authorities. Municipalities derive their power from the provinces.

The PPRE builds on the federal role to address emissions nationally (through a group such as the FCM) and to target regions where cross-border air emissions are an issue (e.g., southern Ontario). However, the specific roles for working with municipal fleet operators and funding retrofits to reduce emissions from diesel buses and heavy-duty trucks is more strongly aligned with provincial and municipal roles. This is evidenced by the existence of similar provincially funded programs in British Columbia and Ontario (see Annex 2 for details). In fact, the literature review, in comparing the PPRE to other relevant capacity-building programs in Canada, highlighted that these programs were mostly designed for a regional audience and implemented at the regional and local levels by NGOs.

Views from interviewees indicated that this kind of pilot was unique for Environment Canada to undertake, as complementary measures to regulations are of interest to the Department but not its primary focus. Stakeholders believed that the appropriate federal role is primarily to regulate air quality, and secondarily to provide national capacity building in order to support municipalities as well as to provide incentives for change (adoption of new technology).

Top of Page

4.5 Performance – Design and Delivery – Performance Information

Evaluation Issues (and related questions)

Indicators

Overall Rating

4. Are appropriate performance data being collected, captured and safeguarded? If so, is this information being used to inform senior management / decision makers?

  • Existence of effective performance measurement plan
  • Extent to which performance data are reliable, timely and relevant
  • Extent to which performance data inform/support Environment Canada’s decision-making processes

Achieved

Summary

The evaluation found that a performance indicator was included in the original program design for the PPRE, but has yet to be reported on. Performance information has been provided by funded partners, primarily at the output level, in a timely and relevant manner. However, there is little evidence of a formal internal performance reporting mechanism to senior management and/or federal partners for the entire pilot project. As the pilot has ended, performance data are not required to support future decisions, but it is relevant to determine whether the pilot has achieved its expected outcome. As it is expected that Environment Canada / FCM will report on the stated performance indicator at the end of the pilot, and other performance data have already been reported, the overall rating has been assessed as Achieved.

A comprehensive performance measurement plan was not developed for the PPRE. Rather, the program design document presented the following anticipated results and performance indicator for the pilot project:

Although this level of detail was considered appropriate by Treasury Board to report on this initiative, no performance information on the defined indicator has yet been compiled. However, the FCM is conducting a final survey of all participants within its Enviro-Fleets Pilot Project, which will produce the required performance information for this indicator.

The work conducted by the FCM under the PPRE had a number of defined performance measures to track outputs and outcomes. These were reported on regularly within its quarterly reports. In addition, the FCM is planning to conduct an end-of-pilot survey with municipalities to better assess the impact of the pilot project. This information was not available for the evaluation but is expected by the end of March 2011. The performance information provided by the FCM thus far was assessed to be reliable, timely and relevant to the project.

The other initiatives funded under the PPRE (i.e., CAP, MSC, Bronson Consulting study) did not have specific performance measures, but were required to submit deliverables as part of their contracts. Interviews with program staff confirmed that all deliverables were completed appropriately and in accordance with the terms and conditions presented in the contracts between Environment Canada and those who received PPRE funding.

Although performance information was collected and reported by the FCM, and contract deliverables were produced as planned, there is no evidence of formal internal reporting mechanisms to senior management or other federal partners on the status of the pilot project, including all of its activities (i.e., those completed by the FCM, CAP, MSC and Bronson Consulting). Program managers indicated that they shared the PPRE outputs and results with management and partners informally through meetings/briefings or by distributing the final deliverables to relevant colleagues.

Top of Page

4.6 Performance – Design and Delivery – Program Design

Evaluation Issues (and related questions)

Indicators

Overall Rating

5. Is the program design appropriate for achieving expected program results?

  • Plausible link between program activities, outputs and intended outcomes
  • Clearly defined and understood governance structure, including program processes, roles, responsibilities and accountabilities
  • Program resources/capacity commensurate with expected program results
  • Views on the appropriateness of program activities, processes and governance structures

Achieved

Summary

The evaluation found a plausible link between program activities, outputs and expected results for the pilot project. There was a clearly defined, understood and appropriate program design and governance structure for managing projects, including their deliverables, timing, funding, and reporting requirements. In addition, program resources and the capacity of internal staff and funded partners were commensurate with the expected results. Therefore, the overall rating has been assessed as Achieved.

The program design document is the foundation of the PPRE design and delivery. It clearly lays out the rationale; anticipated results and performance indicator; activities, deliverables, timelines and funding; and the management of the projects and requirements for evaluation. The first year of the pilot was spent consulting with partners and determining the most appropriate activities to be undertaken, not only to meet the goals but to fill necessary gaps. The program design document was produced after this first year. Interviews indicated that it was important to take this time to investigate what had been done, what was being done by others, and what was needed to ensure the PPRE could provide valuable outputs.

As a result, the PPRE was appropriately designed to use the available resources and capacity of funded partners, as expert delivery agents, to achieve the intended results. Although no logic model was developed, the link between the planned activities, outputs and intended outcomes was plausible.

Although there was little formal management or governance structure for the pilot, this is seen as appropriate given the small amount of funds, the oversight provided by the Strategic Transportation and Policy and Analysis Section, and the controls put in place to effectively manage the funded partners (e.g., via contracts or contribution agreements).

Top of Page

4.7 Performance – Design and Delivery – Barriers

Evaluation Issues (and related questions)

Indicators

Overall Rating

6. Are there any barriers or challenges that could affect the success of this project if it were implemented nationally or integrated into an existing national project?

  • Views of stakeholders and project managers
  • Evidence of documented barriers or challenges
  • Evidence of solutions or approaches to addressing barriers or challenges

Achieved

Summary

The evaluation found that the key barriers to implementing a national program to address HDD emissions resulting from municipal and school bus fleets are: a) funding; and b) ability to work with all components of a municipal / school bus fleet (sometimes composed of both public and private vehicles). There were specific challenges identified in component pieces of the pilot that could be addressed by a broader, national strategy, but some elements, such as determining the amount of GHGs produced by HDD vehicles within a municipal / school bus fleet, are difficult to determine due to the heterogeneous composition of municipal / school bus fleets. As the barriers were clearly identified, and some recommendations for how to address them were included in component reports, the overall rating has been assessed as Achieved.

Two reports produced from the PPRE identified specific challenges that should be considered in developing a broader program to reduce emissions from on-road municipal and school bus fleets. The Bronson Consulting report noted that any clean air program for school buses should be part of a comprehensive school bus health and safety program in order to achieve high success rates, and should be linked with a larger national strategy. Implementation challenges included operators’ knowledge gaps regarding emission issues, and funding constraints. The report noted that program delivery agents from various sectors identified that the optimal role for the Government of Canada in a national clean air for school buses program is that of facilitator rather than being the driving force.

The MSC report on school bus retrofit programs noted that the key barrier is funding, and that another barrier is the reluctance of some school bus companies to consider diesel retrofit technologies for their buses due to time constraints and, in some cases, confidence in specific technologies. Some operators believe that the availability of new technologies, and legislation requiring the transport sector to reduce diesel emissions from school buses over the next several years, will significantly reduce air pollutants and that other efforts are therefore not necessary.

All stakeholders interviewed believed that funding was the key constraint in rolling out a more comprehensive national program. In addition, the variety of players that make up municipal / school bus fleets (e.g., vehicles owned and operated by municipalities vs. vehicles subcontracted by municipalities), along with the variety of information sources and technologies that currently exist, were seen as barriers to action. Busy fleet managers need one integrated and easy-to-access source of information to facilitate adoption. Also, multiple jurisdictions need to be engaged to establish a longer-term and more comprehensive approach.

Top of Page

4.8 Performance – Design and Delivery – Communications

Evaluation Issues (and related questions)

Indicators

Overall Rating

7. Have the results of the pilot project been communicated effectively? How could communications be improved?

  • Views of project managers, partners and recipients
  • Extent to which planned communications activities have been implemented as intended
  • Evidence of information sharing, distribution of project publications and reports, etc. to public fleet operations and other fleets

Achieved

Summary

The evaluation found that there was extensive communication and information sharing with the target audience (municipal fleet operators) to promote and facilitate emission reduction actions, particularly as part of the FCM project, with FCM leading the majority of the communication and information-sharing activities. Therefore, the overall rating has been assessed as Achieved. It was noted, however, that information sharing within the federal community was informal and at times inconsistent.

 

Table 8 summarizes how the FCM and CAP have shared information with organizations.

Table 8: Evidence of Information Sharing by FCM and CAP as part of the PPRE

Delivery Organization
and Information Shared


Number

FCM

Number of surveys completed by municipal fleets

21

Number of inquiries received from municipalities seeking support on municipal fleets

50

Number of participants attending the webinars and workshops

207*

Number of municipal contacts in the network that was established

324

Number of contacts on the list generated by the FCM’s ClearVantage that were sent information on resources and educational opportunities**

1300

Number of resources guides distributed

60

Number of municipalities responding to impact survey planned for March 2011

Figures not yet available

CAP

Number of organizations to which the report was sent directly

10

Number of views on the CAP web page dedicated to the school bus project (e.g., views of fact sheets)

1146 page views***

* 140 municipal representatives and 67 non-municipal representatives. Does not include participants who attended the FCM’s EnviroFleets Workshop, who were not registered; estimated at 50 participants.
** Information was also sent to subscribers to the FCM weekly newsletter and Partners for Climate Protection members.
*** 818 views from unique IP addresses were reported.

 

In addition, Environment Canada indicated that the Department distributed project outputs (e.g., Bronson Consulting report, Clean Air Agenda’s Clean Air Partnership Report) to other federal partners (Transport Canada and NRCan) as a means of sharing information. However, some interviewees stated that they had not been informed of some deliverables, such as the program design document or FCM publications such as the best practices guide.

Top of Page

4.9 Performance – Design and Delivery – Best Practices

Evaluation Issues (and related questions)

Indicators

Overall Rating

8. What are the best practices and lessons learned as a result of the project?

  • Identified lessons learned and shared best practices among partners
  • Identified strengths and weaknesses
  • Views on strengths / best practices, weaknesses and lessons learned

Achieved

Summary

The evaluation found that best practices and lessons learned were provided as part of the component pieces of the pilot. A best practices guide was produced for municipal fleet managers, and advice was provided for addressing emissions from school buses. Therefore, the overall rating has been assessed as Achieved.

Elements of the pilot project identified best practices, lessons learned and/or challenges as a result of the pilot activities. The FCM produced a best practices guide that aims to help municipal fleet managers find cost-effective ways to reduce emissions from heavy-duty vehicles. The guide focuses on best practices relative to overall fuel use (e.g., life cycle management, idle-reduction technologies, etc.). The FCM also produced a resources guide to help fleet managers access the resources they need to improve their fleet efficiency. The CAP, MSC and Bronson reports completed under the pilot provided advice and recommendations for moving forward in addressing CAC emissions from school buses.

In terms of moving forward from this pilot, a summary of the best practices and lessons learned is provided in Section 6 of this evaluation.

Interviewees highlighted that it is important to take time at the beginning of a new pilot project such as this to investigate what others are doing, and what is needed, to ensure the pilot/program meets a clear need and builds on existing programs and initiatives. Many indicated that pilot projects such as this are important for demonstrating what can be done with a finite amount of resources.

Top of Page

4.10 Performance – Effectiveness – Outputs and Outcomes

Evaluation Issues (and related questions)

Indicators

Overall Rating

9. To what extent have the intended outputs and outcomes been achieved as a result of the project?

  • Views of project managers, partners and recipients on achievement of outputs and outcomes
  • Analysis of available performance data on the project’s expected results and performance indicator
  • Population-weighted count of municipalities through FCM that report implementing green fleet initiatives

Progress Made; Attention Needed

Summary

The evaluation found that all but one of the planned outputs have been achieved under the pilot project (the FCM did not obtain sufficient responses to establish a baseline of municipal HDD fleets across Canada). There was a consensus from the evidence and interviews that the pilot project achieved its intent. As the pilot recently ended, it is too early to determine the extent to which this intended outcome has been achieved, although the evaluation did find some evidence to suggest that progress is being made. Information on the performance indicator “population-weighted count of municipalities through FCM that report implementing green fleet initiatives” is not yet available, although it is anticipated that this information will be received through a final assessment survey to be carried out by the FCM in March 2011. Therefore, the overall rating has been assessed as Progress Made; Attention Needed.

Table 9 summarizes the planned and delivered outputs under the PPRE pilot project. It is evident from this table that nearly all of the planned outputs were delivered.

Table 9: Planned and Delivered Outputs for the PPRE

Planned Outputs

Delivered Outputs

FCM

Establish partnerships required to deliver an effective program that will meet the goal. Examples include municipal fleet managers’ networks, federal and provincial agencies, environmental NGOs and private sector organizations.

Create a baseline of municipal heavy-duty fleet within Canada including a fleet profile.

X

Overview of best practices and identification of barriers for greening fleet initiatives.

Identify and create an online resources guide for municipal fleet managers that will provide information on best practices available to achieve emissions reductions.

Promote the resources guide and other products and services.

Deliver two webinars and four education and training workshops in collaboration with partners, using resources identified to increase the capacity of municipal fleet managers to act.

Create a resources guide for reference and use by municipal fleet managers.

Develop a business case for emissions reductions initiatives in heavy-duty fleets, for inclusion in the resources guide.

CAP

Develop a brief report that:

  1. outlines steps taken to collect information on the demographics of school buses in Ontario, and summarizes the data collected;
  2. outlines the steps taken to determine familiarity with, and action on, the 2005 Ontario Public Health Association report School Buses, Air Pollution & Children’s Health and/or its findings, among key staff within school boards, school bus operators, the public health sector, and health associations; and summarizes the findings;
  3. outlines the steps taken to research the language being used in school board policies and/or contracts related to school bus emissions in Ontario, British Columbia and Nova Scotia, and summarizes the findings;
  4. includes model language that can be incorporated into school board policies and/or contracts, related to school bus retrofits and other emission reduction options, for use in Ontario and across Canada.

Develop a four-page fact sheet on school bus retrofits, directed at school board staff and to be disseminated across Ontario.

Develop a four-page fact sheet on school bus retrofits, directed at school bus operators and to be disseminated across Ontario.

My Sustainable Canada

Investigate diesel retrofit technologies to determine which technology is most suitable for use in school buses, and test it in a pilot program for school buses in a community in northern or eastern Ontario.

Bronson Consulting

Develop a report that features a review of school bus characteristics and North American school-bus-focused clean air programs, provides an understanding of how clean air programs could affect the Canadian school bus sector, and summarizes the key considerations from these programs.

 

The anticipated result/outcome for the first initiative under the pilot project was: “Increased implementation by Canadian municipalities of specific actions to reduce emissions from municipal on-road heavy-duty diesel vehicles,” with the following performance indicator: “Population-weighted count of municipalities that participate or report through FCM that they are implementing greening fleet initiatives consistent with the pilot project (i.e. first initiative).” As the pilot has just ended, the degree to which the anticipated outcome has been achieved is not yet documented. The FCM will be conducting an end-of-pilot survey of municipalities that is expected to provide information to address this outcome. In addition, future FCM reports (expected circa 2012-2013) will provide information on how municipalities took actions with their on-road HDD vehicles (e.g., new applications to the FCM’s Green Municipal Fund and its Partners for Climate Protection annual measures report on actions taken related to the HDD fleet).

The following information indicates that progress has been made toward achieving the stated outcome:

Top of Page

4.11 Performance – Effectiveness – Intended Outcomes

Evaluation Issues (and related questions)

Indicators

Overall Rating

10. Have there been any unintended outcomes, either positive or negative?

  • Presence/absence of unintended outcomes
  • Views on whether unintended outcomes occurred

N/A

Summary

The evaluation found no negative unintended outcomes.

 

There was no evidence of unintended outcomes identified in the documentary evidence. Interviewees noted the following two key unintended outcomes:

Top of Page

4.12 Performance – Efficiency

Evaluation Issues (and related questions)

Indicators

Overall Rating

11. Has the project undertaken its activities and delivered products (e.g., technical studies) in the most efficient manner?

  • Analysis of project operational costs in relation to the production of outputs
  • Views on whether the cost of producing project outputs is as low as possible
  • Evidence of / views on whether there are alternative, more efficient ways of delivering project activities and outputs

Achieved

Summary

The evaluation found evidence that Environment Canada partnered with other organizations to achieve the program’s objectives in an efficient manner while maximizing limited resources to manage the program within the Department and leveraging funding from partner organizations (i.e., the FCM).

Therefore, the overall rating has been assessed as Achieved.

Table 10 presents the expenditures of the PPRE against original allocations. These expenditure figures were provided by program staff.

Table 10 PPRE Allocations and Expenditures

The financial data provided highlight that $116,664 in G&C funding available for the pilot project’s first initiative in 2009-10 was not spent. Program staff explained that this funding was not provided to Environment Canada until late in the fiscal year 2009-10 (released in September), and therefore the program partner (FCM) only had a limited part of the year to spend the contributions effectively. As a result, some of the allocated funds available to the pilot project were lapsed.

In terms of comparable federal government approaches to reducing diesel emissions, the literature highlighted that the United States maintains a more comprehensive approach to delivering its objectives concerning diesel emissions, through the U.S. EPA’s National Clean Diesel Campaign (NCDC), which provides access to approximately $200 million (between 2007 and 2011) in funding to manufacturers, fleet operators, air quality professionals, environmental and community organizations, and state and local officials, to reduce diesel emissions (see Annex 2 for details). In the absence of such a large funding envelope to address diesel emissions in Canada, the PPRE utilized partnerships with not-for-profit and/or expert organizations to achieve the project objectives.

The evaluation highlights that the PPRE was implemented by funding not-for-profit and/or expert organizations to undertake specific studies and initiatives. The funded partners were chosen based on their expertise in the required area and their ability to deliver work effectively and efficiently. For example, the FCM was selected as an appropriate organization to work with based on their ability to access municipalities and fleet managers through the FCM’s existing networks. The funds provided to these groups were relatively small (ranging from $25,000–$170,000) and all the deliverables were produced on time and met the Department’s expectations. The project was managed efficiently by one FTE in Environment Canada and was designed to complement existing initiatives.

In addition, some leverage was achieved through the FCM initiative (e.g., $37,000 was provided to the project by the Green Municipal Fund, which resulted in a 14% increase in the original $230,000 in G&Cs allocated for the project). Leverage was also achieved by the FCM collaborating with Fleetwiser to co-promote projects, with FPInnovations, and with Fleet Challenge Ontario (FCO) to deliver the Ontario workshop (FCO contributed $1,000).

Interviewees noted that the funded projects and the work managed by Environment Canada were undertaken in an efficient and collaborative manner. For these reasons, the PPRE was assessed to be efficient.

Top of Page

4.13 Performance – Economy

Evaluation Issues (and related questions)

Indicators

Overall Rating

12. Has the project achieved, or is it on track to achieving, its intended outcomes in the most economical manner?

  • Views on whether good value is being obtained with respect to the use of public funds
  • Evidence of / views on whether there are alternative program models that would achieve the same expected outcomes at a lower cost

Achieved

Summary

The evaluation found that the project is on track to achieving its intended outcomes in an economical manner. All stakeholders viewed the PPRE as a good use of public funds, and believed that the delivery approach was economical. Therefore, the overall rating has been assessed as Achieved.

As noted above, the delivery approach was seen as economical, as it used expert non-profit groups that completed the required tasks for a relatively low cost and in a timely manner. The FCM project was able to reach a wide audience across Canada, to ensure a strong and equitable reach.

Interviewees believed that the federal government, on its own, could not have completed the work in the time required and for the available budget. For instance, it would not have been able to reach the target municipalities economically without the FCM, which had a pre-existing national network and channel in place. The FCM also had established credibility with the target audience, facilitating uptake and engagement. In addition, the FCM built on existing regional initiatives in British Columbia, Quebec and Ontario, and leveraged its broader networks and resources through the Partners for Climate Protection program and Green Municipal Fund. One alternative suggested for enhancing the economy and reach of the FCM workshops was to implement a train-the-trainer approach rather than hold individual workshops.

As well, alternatives could have been used to conduct the studies completed by MSC, CAP and Bronson Consulting, but these would likely not have been as economical, as these groups had successfully and recently conducted previous related studies and were able to build on this work in an economical and timely fashion. One interviewee noted that an alternative such as funding municipalities to do outreach or retrofits may be problematic, as they may not have the same drivers for action (they may be more concerned about fuel savings than environmental benefits).

The literature review (see Annex 2) indicated that the PPRE’s delivery is consistent with the delivery of regional programs that are being implemented in Ontario, British Columbia and Nova Scotia. Key highlights include:

The literature review also demonstrated that there are multiple options for developing capacity with the goal of reducing HDD emissions, including the following:

 

5 2008 Speech from the Throne

 

Previous page | Table of Contents | Next page