Evaluation of Canada's Participation in the Commission for Environmental Cooperation (CEC)

Annex 2 - Evaluation Issues and Questions⁸⁸

Questions	Statement of what should be observed	Indicators	Sources/Methods
Issue: Relevance
1. Does the CEC contribute to the overall federal government agenda?	CEC raison d’être supports the federal government agenda.	Federal government objectives CEC mission statement and its aspects Evidence of North American trilateral cooperation need Government of Canada efforts to review CEC workplans and outputs in relation to federal government priorities	Review of CEC documentation Speech from the Throne (SFT) & other relevant federal speeches, including those on trilateral cooperation Federal Budget Key informant interviews (KII) KII-1 (see Section 5 of Evaluation Plan for group)
Issue: Success Focused on whether and how Canada has benefited from, contributed to, or been impacted by CEC’s efforts to deliver NAAEC objectives in following areas:
Policy Development
2. Have CEC &/or relevant Canadian actions/efforts resulted in:	As a result of these actions/efforts:
a) Mutually supportive environmental and economic policies among the Parties [NAAEC Article 1(b)]?	Relevant policies between Canada and other Parties are harmonized.	Degree of harmonization of policies across Parties Evidence of CEC activities striving toward harmonization	Review of CEC documents Review of Canadian policy documents KII-1 and KII-2
b) Strengthened cooperation regarding the development and improvement of environmental laws, regulations, procedures, policies and practices involving Canada [NAAEC Article 1(f)], including the promotion of economically efficient and effective environmental measures [NAAEC Article 1(i)] and of pollution prevention policies and practices [NAAEC Article 1(j)]?	Development and improvement of relevant environmental laws, regulations, procedures, policies and practices involving Canada.	Evidence of development of, and/or change or improvements to relevant laws, regulations, etc. Evidence of impacts on Canadian decision-makers Number of relevant reports produced by CEC, EC, etc. Relevant Canadian policy objectives Presence of pollution prevention policies and practices promoted by CEC Presence of links between content of CEC reports and relevant pollution prevention policies and practices involving Canada	Review of CEC reports and others including OECD ones Secretariat Reports (NAAEC Article 13) SFT & Budget 2006 Comparative analysis between CEC reports and reports from other sources Review of relevant EC policy documents (e.g., Report on Plans and Priorities (RPP) and GoC state of the environment reporting initiatives KII-1 and KII-2 Review of Taking Stock reports Review of Public opinion research (POR)
c) Enhancement of compliance with and enforcement of environmental laws and regulations involving Canada [NAAEC Article 1(g)]?	Compliance and enforcement of environmental laws and regulations involving Canada have been enhanced.	Presence of more effective compliance and enforcement mechanisms (see NAAEC Art. 5 for example) Evidence of Canadian actions/efforts to ensure compliance and enforcement is occurring	Review of response to CESD Petition 166 Review of Submissions on Enforcement Matters (NAAEC Art.14) Review of Factual Record (NAAEC Art. 15) KII-1 and KII-2 Review of EC documents Review of publicly released non-compliance information
Links to NAFTA
3. Has the CEC supported the environmental provisions of the NAFTA [NAAEC Article 1(d)], including the avoidance of creating trade distortions/barriers [NAAEC Article 1(e)]?	The CEC Council has cooperated with the FTC in an ongoing fashion to achieve the environmental goals and objectives of NAFTA by acting as a point of inquiry, providing assistance in consultations, and contributing to the prevention or resolution of environment-related trade disputes and/or trade distortions/barriers involving Canada.	Number of meetings between Council representatives and FTC Number of meetings between trade and environment officials of the Parties Presence of requirements, strategies and/or events supporting cooperation between the FTC and CEC Presence of relevant trade distortions/barriers averted due to CEC actions/efforts. Number of recommendations to the FTC with respect to avoidance of trade distortion/barriers Number of experts identified to provide information or technical advice for the purpose of contributing to dispute prevention or resolution Specific case examples involving Canada	Review of relevant Council activities(as per NAAEC Art. 10(6)) Review of policy creation or adjustment Review of the NAFTA, NAAEC, CEC Strategic Plan 2005-2010 and the Analytical Framework for Assessing the Environmental Effects of the NAFTA Review of related submissions to the CEC Review of key CEC documents KII-1 and KII-2 Review of key relevant Canadian trade disputes and/or trade distortions/barriers. Review of relevant trade statistics and/or Canadian trade reports
Cooperation
4. Has Canada increased its cooperation with other Parties to better conserve, protect, and enhance the environment as a result of CEC efforts/actions [NAAEC Article 1(c)]?	Canada has increased its participation in relevant cooperative efforts.	Number of agreements Number of conferences Joint publications	Review of agreements Review of conference documentation Review of joint publications KII-1 and KII-2 Federal speeches/documents on trilateral cooperation
Public Participation
5. Did the CEC promote transparency and Canadian public participation in the development of environmental laws, regulations and policies [NAAEC Article 1(h)]?	Active Canadian public participation and timely and relevant responses/actions by the CEC and/or Canada.	Number/scope/nature of submissions, including Canadian ones Instances of informing the Canadian public regarding relevant activities and environmental performance Presence/absence of responses/actions taken by the CEC (from Canada or EC) in response to public submissions Evidence of Canadian cooperation efforts in the submission process	Review of submissions on enforcement matters and factual records (NAAEC Art.14 &15) Review response to CESD Petition 166 Review of responses to and actions taken by CEC in response to submissions KII-1 and KII-2 Review of relevant EC actions Review of relevant public reporting tools (e.g., NPRI)
Unintended Outcomes
6. Were there any unintended outcomes (positive or negative)? If so, how were they addressed?	Unintended outcomes are understood and needed actions are undertaken.	Presence of unintended outcomes Management actions	Review of planning documents, correspondence KII-1 and KII-2
Issue: Design and Delivery Focused on the impacts of the functioning of key bodies and activities, performance monitoring, and Canadian stakeholder groups’ ‘satisfaction’
7. Are the functions/activities of the CEC delivered as designed?	CEC functions/activities are consistent with and follow the CEC’s defined approach.	Actions implemented against planned initiative design	Review of annual reports, IRC of the NAAEC, TRAC report KII-1
8. How well are the main CEC sub bodies working together (Council, Secretariat, Joint Public Advisory Committee (JPAC))?	Sub bodies have clearly defined roles and responsibilities and are working together effectively through, for example, the effective transfer of knowledge and information and without any duplication of efforts.	Defined roles & responsibilities of respective sub bodies Observance of defined roles & responsibilities Mandates of sub bodies Presence of overlap or complementarity in activities Knowledge and information transfers across sub bodies	Documentation review KII-1
9. Is performance data collected against CEC activities/outcomes? If so, is collected information used to inform CEC sub bodies/Parties and relevant to Canada?	Performance data is collected against CEC activities/ outcomes and is used to inform decision making. Collected performance data is also relevant to Canada.	Presence of performance data system collected by the CEC Evidence of performance monitoring Decisions based on performance information Utilization of CEC performance data by Canada Compatibility of CEC and EC performance data systems	Review of CEC planning documents, decisions Review of performance data documentation KII-1 and KII-2 Review of EC performance data systems
10. Is the CEC reaching its Canadian audience (i.e., GoC officials, ENGOs, Canadian public, Canadian trade officials, others)?	CEC activities are directed at and received by Canadian audience.	Reach of target audience (web visits, news stories, number of submissions) Evidence of impacts on Canadian decision-makers	Analysis of reach approaches (e.g., outreach activities, website, reports) Review of submissions Review of policy docs KII-1 and KII-2
11. To what extent are Canadian stakeholders satisfied/dissatisfied with the CEC (e.g., ENGOs, academic & business communities, other organizations, aboriginal)?	Expressed verbal or written satisfaction/dissatisfaction of key stakeholders.	Documentation indicating satisfaction or dissatisfaction	Review of relevant documentation Media scan KII-1 and KII-2
Issue: Cost-Effectiveness/Alternatives Are the most appropriate, cost-effective and efficient means being used to achieve objectives as they relate to Canadian needs?
12. Are other national or international organizations involved in the same areas of activities and/or do they share similar objectives as the CEC? What is Canada’s role in these organizations?	CEC objectives and related activities are different from and/or complementary to the areas of activities/objectives of other organizations. Canada’s role in these organizations is also well-defined.	Absence/presence of duplication in activities/outcomes Absence/presence of similarities/differences in activities/outcomes Presence of Canadian participation in other organizations	Review of relevant organizations’ (e.g., OECD) and CEC documentation KII-1 and KII-2 Analysis of relevant Canadian international activities
13. Are there any high level preoccupations and potential risks related to the administrative and financial policies of the CEC and its organizational structure, especially in regard to some issues expressed by key CEC stakeholders?	CEC administrative and financial rules, policies and procedures appear to be exhaustive.	CEC has rules, policies and procedures covering all major administrative and financial functions	Interviews with CEC staff Review of the inventory of CEC rules, policies and procedures
	CEC financial budgeting and reporting appear to be transparent and conducive to accountability.	The level of information provided through CEC budgeting and reporting is more transparent and conducive to accountability in comparison to concerns expressed by key CEC stakeholders in 2005	Comparison of the level of information provided for budgeting and reporting between financial years 2005 and 2006 Benchmarking with one comparable organization Interviews with CEC staff and NAAEC Parties
	Travels are planned in advance and are subject to a transparent and adequate approval process.	Travels are planned in advanced A transparent and adequate process is in place to approve travels	Interviews with CEC staff Review of CEC rules, policies and procedures
	Contractor support services are subject to a transparent and adequate approval process.	A transparent and adequate process is in place to approve Contractor support services	Interviews with CEC staff Review of CEC rules, policies and procedures
	The organizational structure is conducive to effective delivery of results.	Ratio of management level resources versus staff	Benchmarking with one comparable organization

⁸⁸ The evaluation issues and questions were taken from Table 1 in the Evaluation Plan for the Evaluation of Canada's Participation in the CEC, September 2006. We note here that modifications to this Table were made over the course of the evaluation, for example, to better reflect appropriate information sources and/or indicators, etc.

Previous page | ToC | Next page

Evaluation of Canada's Participation in the Commission for Environmental Cooperation (CEC)

Annex 2 - Evaluation Issues and Questions88

Annex 2 - Evaluation Issues and Questions⁸⁸