Top of Page
Previous page | ToC | Next page
The purpose of the evaluation was to examine Canada's participation in the CEC, in particular, by assessing whether and how Canada has benefited from, contributed to, or been impacted by CEC's efforts to deliver NAAEC objectives.
The evaluation consisted of a comprehensive examination of the following areas: key CEC substantive activities, governance aspects related to the CEC and the Canadian government, and CEC operational and organizational aspects, including CEC administrative and financial processes. Close attention was given to the role played by EC as it is the federal department accountable for the financial contributions provided to the CEC.
In assessing Canada's participation in the CEC, the following four issues were used to guide the evaluation:
Note that no documentation describing expected outcomes (and associated activities and outputs) related to Canada's participation in the CEC was developed. EC's past contribution to the CEC has been exempted from the development of a Results-based Management and Accountability Framework (RMAF).21 Canada, however, participates in the CEC as it is the organization which was created to facilitate the effective delivery of the NAAEC's objectives. The assessment of the evaluation issue of success was hence based on these objectives which are presented in Annex 1.
The objective areas mentioned above under the evaluation issue of success represent the evaluation's conceptual summary of the NAAEC objectives. In regard to the area of policy improvement, six of the ten NAAEC objectives refer to this area in one form or another (e.g., measures, laws, regulations, procedures and practices), including the promotion of mutually supportive environmental and economic policies. The areas of environmental cooperation, public participation and trade and environment were included given their direct and indirect links to specific NAAEC objectives. The trade and environment area was also included in the examination of the evaluation issue of success in light of the previously mentioned importance given to this area in past and current CEC work and activities.
The specific questions pertaining to each evaluation issue indicated above are presented in the Evaluation Plan for the Evaluation of Canada's Participation in the CEC (September 2006). The details of these are found in Annex 2. The evidence from this evaluation was gathered between July 2006 and February 2007.
In order to support the evaluation process, an evaluation committee was created. This committee was composed of officials from EC's Audit and Evaluation Branch (the evaluation project team) as well as program managers involved in CEC activities at EC and DFAIT. The mandate of the evaluation committee was to facilitate and guide the evaluation process at the working level from start to finish.22
An important part of the evaluation was to share preliminary findings and draft reports with individuals, groups and/or committees to allow them to validate findings and/or provide feedback before the report was completed. In this context, separate sessions were conducted with the following groups: evaluation committee members, members of EC's Strategic Integration Board (comprised of senior EC officials), CEC Secretariat senior staff, representatives of the signatory provinces, and Alternate Representatives and officials from the other two NAAEC Parties, namely Mexico and the U.S.23
As part of the Audit and Evaluation Branch's internal quality assurance process, copies of the draft report were circulated to the Branch's senior management for a first review. The draft report was then distributed to the evaluation committee members for their review. Finally, the draft report was circulated to CEC Secretariat senior staff as well as to representatives of the signatory provinces and of the U.S. and Mexican governments.
In accordance with best practices, the evaluation involved the use of multiple lines of evidence and modes of enquiry, including:
Documents pertaining to both the Government of Canada and the CEC were reviewed and analyzed. A full list of these documents and files can be found in Annex 4. In addition to the policy, planning and research-related documents regarding both the CEC and Canada's participation, this list also contains the key CEC rules, policies and procedures that were reviewed by the evaluation team.24
Monitoring of Canadian mainstream media provided information about the different perceptions that people/groups may have about the CEC. The evaluation used both an EC-based media research tool as well as the media and outreach impact reports that are produced by the CEC Secretariat.
Key informant interviews were conducted with Government of Canada senior and junior level officials (with past and ongoing dealings with the CEC), CEC Secretariat senior staff (past and existing), members of the Canadian JPAC and NAC (past and existing), and representatives of the signatory provinces (generally officials of the Governmental Committee). These interviews, the majority face-to-face, were conducted by the evaluation project team officials between September 20th, 2006 and October 20th, 2006. A total of 39 interviews were completed. The purpose of these interviews was to examine a number of issues and questions covered in the present evaluation.25 Annex 5 presents the interview questions and themes that were employed to facilitate interviewee input.26
The following presents the number of completed interviews of the aforementioned groups:
Key informant telephone interviews with Canadian stakeholder groups were also conducted by senior researchers of Environics Research Group. These interviews were conducted between September 21, 2006 and December 13, 2006. The purpose of this set of interviews was to determine whether and how stakeholders are effectively being reached by the CEC's activities, and the extent to which they are satisfied with the CEC's overall work. Specifically, the stakeholder research explored: 1) familiarity with the CEC, and knowledge of its mandate; 2) opinion of extent to which the CEC is facilitating the delivery of NAAEC's objectives and key factors influencing this, including Canada's contribution; 3) assessment of CEC operations, in terms of the effectiveness and accountability of its three key bodies (Council, Secretariat and JPAC), and the level of cooperation between the bodies; and 4) suggestions/views for the future, in terms of improving the CEC's effectiveness and strengthening Canada's contributions.
The evaluation project team provided Environics with a list of 82 stakeholders who were identified as eligible for inclusion in this consultation. The list of eligible stakeholders was designed to ensure representation across stakeholder groups (to account for differing perspectives) and a balance between those more directly involved and those less involved with the CEC. Interviews were conducted with senior people within these organizations (e.g., Presidents, CEOs, and Executive Directors). A total of 28 interviews were completed in the timeframe available, representing a 34 percent participation rate. This is fairly typical for this type of project involving busy senior professionals, who can be difficult to reach or unavailable for participation due to other commitments. However, the participation level for this particular study was also affected by a number of declines from organizations citing a lack of knowledge about the CEC (this will be discussed in Section 4.2).27
Interviews focused on the following key CEC stakeholder groups (number of completed interviews is presented in brackets):
An interview protocol, developed in conjunction with the evaluation project team as well as members of the evaluation committee was used to guide the interviews. A copy of this protocol is also provided in Annex 5.28
Finally, overall themes were identified for each interviewee groups who participated in this evaluation. These themes taken together with the other lines of evidence underpin the findings presented in this report's Section 4. Where consensus was found, individual interviewee groups are not identified. The report also includes quotes from participants to illustrate highlights that emerged during the interviews.
A review of mandates, activities and operational practices of other comparable organizations was conducted to provide information upon which the CEC may be compared. Given the uniqueness of the CEC (i.e., it cannot be compared to another organisation on all of its components), the evaluation compared the CEC to different organisations on a component-by-component basis. The data gathered for this was primarily based on documents reviewed and the interviews. The following provides examples of key organisations used for comparing and according to which components:
21 An RMAF is a document prepared by managers to help them focus on measuring and reporting on outcomes throughout the lifecycle of a policy, program or initiative. It typically accompanies a Treasury Board Submission.
22 The terms of reference of the evaluation committee can be found in Annex 3.
23 EC's Strategic Integration Board is part of EC's Priority Management Boards. The latter, an integral component of EC's operational governance structures, are responsible for the design and delivery of specific elements of departmental outcomes and priorities. Under EC's Audit and Evaluation Policy, Boards are responsible for the management response to recommendations made in projects. It is the Strategic Integration Board that has been identified as the Board responsible for the management response for this project.
24 The decision to examine CEC administrative and financial processes was based on questions, raised by EC officials and by representatives of the other NAAEC Parties regarding CEC administrative and financial aspects, including financial budgeting and reporting, travel and contractor support services. This part of the evaluation is reflected in question 13 of the Evaluation Plan's Table 1, which may be found in the present report's Annex 2. The evaluation project did not perform any sampling and testing of transactions following the review and analysis of CEC rules, policies and procedures. This piece of the evaluation was therefore intended to observe any potential risks and/or appearance of control weaknesses. The analysis did not cover all administrative and financial processes, such scope being too large to carry out under this project. Finally, we indicate here that, in addition to the validation sessions conducted by the evaluation project team, a follow-up meeting between Audit and Evaluation Branch and CEC Secretariat's Administration and Finance Unit officials (held at Environment Canada) provided an additional opportunity to validate related findings.
25 Once again, we refer the reader to Annex 2.
26 Note that interview guides varied according to the different interviewee group. Annex 5 presents the interview guide for Government of Canada officials as it is the most comprehensive. The other interview guides were adapted to reflect respective needs and knowledge of different interviewee groups.
27 Of the list of 82 stakeholders, 26 stakeholders declined to participate and 17 of these reported that this was due to their lack of knowledge of the CEC. Further details on the final disposition of all stakeholders on the original list provided may be found in the final report prepared by Environics and titled: Canadian Stakeholder Interviews for the Evaluation of Canada's Participation in the CEC (February 2007).
28 In terms of recruitment for all the interviews, EC's Audit and Evaluation Branch officials contacted potential interviewees in advance by e-mail, informing them of the purpose of the interviews and requesting their participation. An official from the Audit and Evaluation Branch (and from Environics for the stakeholder research) subsequently contacted each person by telephone and/or by e-mail to schedule an interview session. Individuals were contacted multiple times, as appropriate, to schedule an interview. Once an interview had been scheduled, the Audit and Evaluation Branch sent the interview questionnaire as well as a confirmation of the date and time of the interview. In the case of the stakeholder research, Environics sent an abbreviated version of the interview protocol to provide an overview of what would be covered.