Home >
Proactive Disclosure >
Audits and Evaluations >
A&E Reports >
2009-2010
> Evaluation of the Enforcement Program
Evaluation of the Enforcement Program
Previous page | ToC | Next page
Annex 1 Evaluation Issues and Questions
Relevance
Does the Enforcement Program remain consistent with and contribute to federal government priorities and address actual needs?
1. Is there a legitimate and necessary role for government in this program area?
What Should be Observed
Indicators
Data Sources/Methods
Existence of private market failure or need to protect a perceived public good
The program is aligned with
EC and federal priorities
- Demonstration of clear mandate that is aligned with a public good
- Demonstration of link of program’s mandate with EC’s jurisdiction
- Consistency of program’s mandate/objectives with current priorities of EC and the Government of Canada
Document Review
- EC Departmental Performance Report (DPR)
- EC Report on Plans and Priorities (RPP)
- Legislation/policies on federal mandate for environmental and wildlife enforcement
- Throne Speech
- Federal Budget
Key Informant Interviews
- Enforcement Program managers and staff
- Internal EC partners
2. Is the Enforcement Program rationale based on actual societal/environmental needs? Does the Program serve the public interest?
What Should be Observed
Indicators
Data Sources/Methods
The program addresses actual needs and serves the public interest
- Demonstration of the utility/rationale and need for the program
- Degree to which the program serves the public interest
- Gaps would exist in coverage without the program
Document Review
- OPP/OPG/Board documentation
- Corporate Risk Profile
- Program documentation and previous reviews
- Treasury Board (TB) submissions
- Public opinion research findings on perceived needs for environmental and wildlife protection and enforcement by the federal government
3. Does the Enforcement Program theory (i.e., objectives, logical linkage of activities and outputs to intended outcomes, instruments used) realistically address the societal needs identified?
What Should be Observed
Indicators
Data Sources/Methods
The program theory is aligned with societal needs/program outcomes
- Soundness of logical linkages between activities, outputs and intended outcomes
Document Review
- Logic model
- TB submissions
Success
Has the Enforcement Program achieved its intended outcomes?
4. To what extent have the intended immediate and intermediate outcomes been achieved as a result of the Enforcement Program?
What Should be Observed
Indicators
Data Sources/Methods
The program shows achievement of (or progress toward) immediate and intermediate outcomes specified in logic model
- Views and documentation on degree of achievement of each intended immediate and intermediate outcome, as specified in logic model
- Trends in regulatee compliance with high priority regulations (analysis provided by Enforcement Program and Compliance Analysis and Planning)
Document/Data Review
- Program documentation on results
- Periodic reports on progress
- NEMISIS and other activity data
Key Informant Interviews
- Enforcement Program managers and staff
- Internal EC partners
- External partners (e.g., OGD, PT governments, international partners)
Case Studies
5. Have there been any unintended outcomes, either positive or negative, that can be attributed to the Enforcement Program? If so, were any actions taken as a result of these outcomes?
What Should be Observed
Indicators
Data Sources/Methods
Unintended outcomes are present that can be attributed to the program
Where appropriate, actions to address unintended impacts were taken
- Presence/absence of unintended outcomes
- Where appropriate, documented management actions and/or lessons learned from unintended outcomes
Document Review
- Program meeting minutes
- Program correspondence
Key Informant Interviews
- Enforcement Program managers and staff
- Internal EC partners
- External partners (e.g., OGD, PT governments)
Case Studies
6. Are there any external factors outside of the Enforcement Program which influence (positively or negatively) the success of the Program?
What Should be Observed
Indicators
Data Sources/Methods
Achievement of program outcomes is influenced by external factors outside the Program
- Identified situations and the degree to which external factors influenced the Program
- Views on the quality of the working relationships between the program and internal and external partners, and how this influences the success of the Program
- Views on the degree of coordination of the priority-setting processes of the program with those of key internal partners (e.g., compliance promotion, Canadian Wildlife Service (CWS), environmental protection regulatory programs), and how this influences the success of the Program
- Views on the degree to which human resources challenges in the Enforcement Branch influence the success of the Program
Document Review
- Program correspondence
- Program meeting minutes
- Program planning documents
Key Informant Interviews
- Enforcement Program managers and staff
- Internal EC partners
- External partners (e.g., OGD, PT governments)
Case Studies
Cost-Effectiveness/Alternatives
Are the most appropriate, cost-effective and efficient means being used to achieve outcomes?
7. Are there alternative ways of achieving the objectives of the Enforcement Program?
What Should be Observed
Indicators
Data Sources/Methods
Alternative delivery methods have been analyzed
- Demonstration of analysis of various delivery options/opportunities
- Lessons learned from alternative approaches in terms of cost-effectiveness
- Views on the degree to which alternative delivery models could improve the Program’s cost-effectiveness
Document Review
- Program/Branch design reports
- Previous program reviews
Key Informant Interviews
- Enforcement Program managers and staff
- Internal EC partners
- External partners (e.g., OGD, PT governments)
8. Are there other programs that are involved in the same activities or that share similar objectives? If yes, how is duplication avoided and complementarity achieved? Are there any gaps?
What Should be Observed
Indicators
Data Sources/Methods
Program complements and does not duplicate other comparable programs
There are no major gaps in activities in this program area
- Views on duplication and complementarity with comparable programs
- Views on presence/absence of gaps
Document Review
Key Informant Interviews
- Enforcement Program managers and staff
- Internal EC partners
- External partners (e.g., OGD, PT governments)
9. How could the efficiency of the Enforcement Program be improved?
What Should be Observed
Indicators
Data Sources/Methods
Program shows opportunity for efficiency improvements
- Analysis of costs for Program delivery (i.e., production of outputs)
- Proportion of total Program budget required for purely administrative activities as compared to proportion for Program delivery (administrative overhead)
- Suggestions for improving the efficiency of the Program
Document Review
- Program design documentation
- Previous program reviews
- Costing details (e.g. salaries, operating costs, administrative overhead)
Key Informant Interviews
- Enforcement Program managers and staff
- Internal EC partners
- External stakeholders (industry associations)
10. Has the Enforcement Program provided value for the federal dollars spent?
The program demonstrates value for the federal dollars invested (
i.e., cost-effectiveness)
Allocation of resources is based on highest
importance and resourced according to priorities
- Degree to which selection process for areas of importance is applied
- Views on the degree to which the Program has provided good value for the federal dollars invested
Document Review
- Program financial information
- National Inspection Plan (NIP)
- National Enforcement Plan (NEP)
Key Informant Interviews
- Enforcement Program managers and staff
- Internal EC partners
Design and Delivery
Is the Enforcement Program designed and delivered in the best possible way?
11. Does the Enforcement Program identify clear deliverables and expected results? Is the Program delivered as designed?
What Should be Observed
Indicators
Data Sources/Methods
Expected results and deliverables are clearly identified
Program delivery is consistent with and follows defined approach
- Demonstration that the Program has clearly defined deliverables and expected results
- Degree to which the Program is implemented as designed, i.e., outputs are produced and delivered to target audiences, as specified in the program logic model (analysis provided by Enforcement Program)
- Views on the strengths, weaknesses and needed improvements to the delivery of the Program
Document/Data Review
- OPP/OPG/Board documentation
- Program plans (e.g., NIP, NEP) and work plans
- Program logic model
- TB submissions
- NEMISIS data
Key Informant Interviews
- Enforcement Program managers and staff
- Internal EC partners
- External partners (e.g., OGD, PT governments)
Case Studies
12. Are performance data collected against Enforcement Program activities and outcomes? If so, is the collected information used to inform senior management/decision makers?
What Should be Observed
Indicators
Data Sources/Methods
Performance data are collected against program outputs and outcomes and are used to inform decision making
- Presence/absence of populated performance data system with reliable and timely data
- Decisions based on performance information
- Views on accuracy and usefulness of existing program performance information and gaps in information
Document /Data Review
- Program planning documents and decision documents
- Performance data/system (NEMISIS) and documentation
Key Informant Interviews
- Enforcement Program managers
13. (a) Who is accountable for the Enforcement Program? Are the roles and responsibilities of the Enforcement Branch clearly defined and implemented as specified? (b) Are the Branch’s roles and responsibilities clearly understood by key internal and external partners and stakeholders?
What Should be Observed
Indicators
Data Sources/Methods
Roles, responsibilities and accountability are clear and duly implemented
- Defined and known program management structure, roles and responsibilities
- Degree to which roles and responsibilities are implemented as specified
- Degree to which partners and stakeholders feel they understand the Branch’s roles and responsibilities
Document Review
- Program documentation and organization chart
- Minutes of program management meetings
- EC Results Management Structure
Key Informant Interviews
- Enforcement Program managers and staff
- Internal EC partners
- External partners (e.g., OGD, PT governments)
Case Studies
14. How effective are the Enforcement Program’s internal and external communications? Are any improvements needed?
What Should be Observed
Indicators
Data Sources/Methods
Internal and external partners feel they receive the essential information they need from the Enforcement Program
Enforcement Program managers/staff feel they receive the essential information they need from internal and external partners
- Satisfaction with the quality and completeness of formal and informal communications between the Program and partners
- Views on the adequacy of feedback mechanisms between the Program and internal EC partners (e.g., Compliance Promotion, CWS, environmental protection regulatory programs) for purposes of strategic planning and coordination of efforts
- Suggestions for improvements to communications
Document Review
- Program communication products and mechanisms
Key Informant Interviews
- Enforcement Program managers and staff
- Internal EC partners
- External partners (e.g., OGD, PT governments)
Case Studies
15. What are the best practices and lessons learned from the Enforcement Program?
What Should be Observed
Indicators
Data Sources/Methods
Identified lessons learned and best practices
Identified strengths and weaknesses
- Identified lessons learned and best practices
- Identified strengths and weaknesses
- Views on strengths/best practices, weaknesses and lessons learned
Document Review
- Program documentation and previous reviews
Key Informant Interviews
- Enforcement Program managers and staff
- Internal EC partners
- External partners (e.g., OGD, PT governments)
Case Studies
16. How effective is the governance of the Enforcement Branch? Are any improvements needed?
What Should be Observed
Indicators
Data Sources/Methods
Governance and consequent activity resulting from governance structure are demonstrated to be effective
- Analysis of effectiveness of decision making processes
- Views on the effectiveness of the governance and organization of the Enforcement Program before versus after the creation of the Branch
- Views on the degree to which regional Branch offices receive the information and direction they need from National Headquarters to perform their enforcement responsibilities
- Suggestions for improving the governance of the Branch
Document Review
- Program organization chart
- Minutes of program management meetings
- Program correspondence
- Program planning documents and decision documents
Key Informant Interviews
- Enforcement Program managers and staff
- Internal EC partners
17. Does the Enforcement Program have adequate capacity in terms of financial and human resources to achieve its intended outcomes and objectives?
What Should be Observed
Indicators
Data Sources/Methods
Program has adequate financial and human resources to achieve its intended outcomes and objectives
- Analysis of financial and human resources (resources allocated and actually received) against capacity requirements to achieve intended outcomes and objectives, overall and by region
- Views on the extent to which the program’s resources are appropriate in consideration of its responsibilities and identified priorities, overall and by region
Document/Data Review
- Program financial information
- TB submissions
- Program planning documents
- Program logic model specifying intended outcomes
- Previous reviews/audits of resource and capacity issues
Key Informant Interviews
- Enforcement Program managers and staff
Case Studies
Previous page | ToC | Next page
