Top of Page
Previous page | ToC | Next page
Environment Canada (EC) committed to Treasury Board (TB) to evaluate the Enforcement Program. This evaluation was part of Environment Canada’s Audit and Evaluation Plan for 2007–2008 to 2009–2010 approved by the Departmental Audit and Evaluation Committee on April 18, 2007. The evaluation was conducted in 2008–2009.
The objective of the evaluation of the Enforcement Program was to assess issues related to the Program’s relevance, design and delivery, success and cost-effectiveness in order to meet the information requirements of senior management at Environment Canada, as well as those of TB. The evaluation covered the four-year time frame from 2004–2005 to 2007–2008, with a particular focus on the period following the creation of the Enforcement Branch in June 2005, while also addressing significant developments to the Program that occurred in 2008–2009.
In addition, the evaluation of the Environmental Enforcement Intelligence Program, previously planned for 2007–2008, was integrated into this broader evaluation of the Enforcement Program15.
The evaluation examined four issues16:
A total of 17 evaluation questions supporting these four broad issues were addressed in the evaluation. In Section 4 of this report, the findings are presented for each evaluation issue as well as for each associated evaluation question. The evaluation questions along with the associated indicators, data sources and data collection methods are listed in Annex 1.
The evaluation involved multiple lines of evidence as described in Table 3.
The evidence for this evaluation relied heavily on qualitative data obtained from key informant interviews, group discussions and case studies. These qualitative methodologies were utilized to obtain views on the evaluation questions from a range of relevant perspectives—including the perspective of the Enforcement Program at National Headquarters and in the regions, key internal partners (the CWS, environmental protection regulatory programs and Compliance Promotion) and key external partners and stakeholders (other federal government departments, provincial governments, international partners and an industry association). Although a qualitative analysis of primary data from the above sources was supplemented with an analysis of secondary data from Program documentation, it was not possible to corroborate respondents’ observations on several of the Program’s results, due to a lack of credible quantitative data or documentation providing direct evidence of Program outcomes. Much of the available documentation provided evidence of activities and outputs only, and it was beyond the scope of this evaluation to undertake new primary data collection for very specific indicators of outcomes (e.g., the level of knowledge and skills of enforcement officers before and after their training, the level of public and regulatee awareness of the responsibilities of the Enforcement Program and benefits of enforcement).
Related to the above limitation, very little performance information was available for the four-year time frame of this evaluation. The Enforcement Branch was able to provide performance information for only a limited number of Enforcement Program outputs (i.e., inspections, investigations and evidence/support for prosecution cases) and outcomes (i.e., investigations leading to prosecutions and successful prosecutions). NEMISIS, the Program’s information system, does not currently have reliable performance data on other outputs and outcomes in the logic model.
In addition, a lack of financial information on planned/budgeted annual expenditures for each year prevented a comparison with the information provided on actual annual expenditures, thereby limiting the analysis of the adequacy of Program resources.18
Finally, it was not possible to draw conclusions on the incremental impacts of the Enforcement Program on the key outcome of regulatory compliance or to distinguish the Program’s contribution to compliance from that of the CWS/regulatory programs or the Department’s compliance promotion function. This limitation was due primarily to a lack of accurate data on compliance rates with major regulations during the evaluation time frame.
15 See footnote 5 regarding the Environmental Enforcement Intelligence Program (Section 1).
16 The planning and data collection for the Enforcement evaluation were carried out in the 2008–2009 fiscal year, prior to implementation of the Treasury Board’s new Policy on Evaluation in April 2009. The current evaluation reflects the issues outlined in Treasury Board’s 2001 evaluation policy, which was in effect at the time this evaluation was conducted.
17 In summarizing the degree of consensus in key informant interview findings, the following guidelines were used: a few interviewees (less than 25%); a minority of interviewees (25% to 44%); approximately half of interviewees (45% to 55%); a majority of interviewees (56% to 75%); most interviewees (76% to 94%); and almost all interviewees (95% to 99%).
18 According to representatives of the Enforcement Program, organizational restructuring and changes in the Department’s financial tracking system made it difficult to keep accurate financial information for the years covered by this evaluation. For some years, departmental budget adjustments and internal allocation decisions made by the CEO created difficulties for tracking the distribution of funding across the Enforcement Program.