What would be the consequences if the program did not exist?6
The program does or does not serve a recognized and needed function.
Previous page | ToC | Next page
Below are the findings of this evaluation presented by the four categories - relevance, success, cost effectiveness and design and delivery—and using the set of questions developed for the evaluation framework.
This section will examine program relevance—the degree of alignment with the directions and priorities of EC and clarity of the intent of GBAP, as well as the continuing rationale for the program given prevailing areas of need and the potential for overlap with other initiatives.
| Q1 Evaluation questions | Statement of what should be observed | Rating3 |
|---|---|---|
| Overall, does the program (and its purpose) make sense in terms of the intent of the CESF [departmental strategic outcomes]? | Program mission/raison d'être supports the intent of the CESF [departmental strategic outcomes]. | √ |
The program documentation as well as staff opinions do indicate a clear connection between the GBAP with the Priority Ecosystems Outcome Project Grouping (OPG) and the Department's broad strategic directions.
The principles outlined by the Department's ecosystem approach are so broad and propose a very non-exclusionary definition which results in a statement that is unconstrained both in space and unbounded by technique. The fundamental design elements of an ecosystem program remain illusive.
| Q2 Evaluation questions | Statement of what should be observed | Rating |
|---|---|---|
| Role of government—Is there a legitimate and necessary role for government in this program area or activity?4 | Demonstrable needs not being met by private sector, and need to preserve as public good. Existence of private market failure or need to protect a perceived public good. |
√ |
The program documentation indicates a role for government (primarily with regard to coordination activities, providing expertise for decision-making purposes, and resources) in protecting and restoring environmental health (seen as a public good); this does not preclude other partners/stakeholders from playing an important role.
Staff opinion indicates the GBAP does serve a public interest, and is, for the most part, responsive to needs and changing needs by complementing other programs and partners' activities. (Some overlap may exist, but no real duplication of effort, except within EC. Some speculated this might be the reason for unbundling.).
| Q3 Evaluation questions | Statement of what should be observed | Rating |
|---|---|---|
Federalism—Is the current role of the federal government appropriate, or is the program a candidate for realignment with the provinces/territories?5 How does this activity or program balance the need for coordinated Canada-wide action with the need for flexibility to reflect the diverse needs and circumstances of provinces/territories and regions? |
The program is situated at the appropriate level of government without need for realignment. |
√ |
Role exists for the federal government, based on documents and interviews.
There were significant cutbacks to provincial ministries whose mandates include the management and protection of key resources, such as the British Columbia Ministry of Water, Land and Air Protection and the British Columbia Ministry of Forests, at least until 2004-2005.
The majority of EC staff interviewed (7 of 12) found it difficult to link GBAP's goals and objectives with the strategic directions and priorities for the Government of Canada, due in most part to the recent change in government; 3 identified air quality objectives as the prime link.
According to the visioning statement for the Priority Ecosystems OPG, ecosystem initiatives are designed to take into account the specific geographical environmental problems that are inherent therein; therefore, each ecosystem initiative is designed to address specific regional environmental issues. However, all ecosystem initiatives have four common principles—they are based on an ecosystem approach; science is used to inform decision making; partnerships are pursued to achieve their goals; and there is a focus on engaging citizens and communities.
| Q4 Evaluation questions | Statement of what should be observed | Rating |
|---|---|---|
What would be the consequences if the program did not exist?6 |
The program does or does not serve a recognized and needed function. |
|
The program is very horizontal in its design: it includes other federal departments, several provincial agencies, and First Nations as partners. Within the Department many different parts of the organization are also involved in the delivery of its projects. Therefore, there is a broad connection to a wide-ranging suite of partners.
The senior EC managers underline the collaborative nature of the program and the public interest being served.
However, the continued cohesiveness of the GBAP program is brought into question through the unbundling process of the Priority Ecosystems OPG. Unbundling masks the specific role of an area-specific ecosystem initiative
| Q5 Evaluation questions | Statement of what should be observed | Rating |
|---|---|---|
Public interest—Does the program area or activity continue to serve the public interest?7 Is the program defined in terms of targeted client groups? |
The program is connected with societal/environmental needs. |
√ |
Public interest is likely to exist; however, no apparent comprehensive and analytical examinations of targeting reach by the GBAP.
There was universal agreement amongst the 22 EC staff and managers that the initiative very much addresses the public interest.
| Q6 Evaluation questions | Statement of what should be observed | Rating |
|---|---|---|
Does the program clearly contribute to delivering departmental outcomes (OPP, OPG) and Board priorities? |
The program is aligned with departmental outcomes and Board priorities. |
√ |
Documentation indicates that there is a clear link between the program and the Priority Ecosystems OPG, Ecosystem Sustainability Board, and departmental outcomes and priorities.
Out of the 12 EC staff responses, 9 had difficulty making a connection with Board priorities (either did not know them, or they were constantly changing); however, the majority were better able to make a connection with departmental outcomes. All Board members (3) felt there was a clear link with the Board priority (focusing on priority ecosystems) and with departmental outcomes. Out of 13 responses, the majority of EC staff felt that the design of GBAP was reflective of departmental priorities although, given changing priorities, some felt this might no longer be the case; all Board members felt the linkage was there as well. Of the 5 EC staff responses, 4 felt that the goals and objectives of GBAP were reflected in project level deliverables (although quality control and enforcement may be an issue).
The OPG does not provide a clear link between the specific accountabilities of other OPPs/OPGs (for which near-term results they are accountable) regarding priority ecosystems and how these accountabilities contribute to achieving the end results of the Priority Ecosystems OPG.
| Q7 Evaluation questions | Statement of what should be observed | Rating |
|---|---|---|
Are changes required to ensure alignment with current departmental priorities as well as the CESF (departmental strategic outcomes)? If yes, is the Program (OPP) and its structure able to accommodate for such changes (e.g. mechanisms). |
Program rationale addresses required changes if needed. |
|
Not all of the 22 EC staff and managers interviewed were of uniform opinion as to need for changes to this program. Some pointed to the failings of a fixed five-year program that is not engaging its partners such as DFO and First Nations sufficiently as the partners (DFO) are withdrawing, weak, or providing insufficient funds. Some questioned the administrative and coordination elements of this program. Others noted the integrative science base of the GBAP and its general overall evolution over time. Some concerns were voiced as to the type and level of reporting requested by the program that staff would find acceptable and reflective of their participation as well as the onerous departmental results structure that does not allow for funds to be moved easily, or expedite approvals. Universally there was no concern voiced as to potential duplication of the GBAP to other programs.
Some of the three senior EC managers questioned unbundling and the capacity to retain the integrative elements. Risks to unbundling are underlined. They stated that there would be a need to ensure that budget allocation is made in an integrative fashion and not just on a single medium approach (air, water, etc). There are implications and connections across OPGs that must be maintained to retain effectiveness; that connection is managed through management committee and through effective partnerships at Board and OPG level.
The four signatory partners vary in their opinions. One states that GBAP is almost invisible for us; it is business as usual. Others state that the environmental aspect is covered by the GBAP but challenges with incorporating the social and economic dimensions remain. Nevertheless, in the opinion of some, progress has been made.
| Q8 Evaluation questions | Statement of what should be observed | Rating |
|---|---|---|
Do all OPPs within the scope of the OPG need to exist? |
There is sound/explicit rationale for all OPPs with regard to the OPG. |
 |
Despite a lengthy history and existence of the individual ecosystem initiative projects, to date, no documented rationale has been developed which clearly identifies the need for each and every GBAP project to exist and indicate their relationship with each other.
There is recognition, however, that a national strategic vision should be developed. Some early discussion has pointed to elements of what that may contain but these have not yet been developed. A definition of an ecosystem approach coupled with a management framework for Priority Ecosystems was scheduled for the fall of 2006. More specifically, the plan for Ecosystem Management was forecast to be in place by April 2006; this has not occurred. The means to identify critical ecosystems was expected to be in place by October 2006 with the Ecosystem Sustainability Board presented these results in November 2006 and the method applied by March 2007.
| Q29 Evaluation questions | Statement of what should be observed | Rating |
|---|---|---|
What are the closely connected existing programs and how is duplication avoided and complementarity achieved (including non-federal government programs)? |
Program mandate or outcomes do not duplicate other programs, or program mandates complement other programs. |
√ |
No comparison document looking at GBAP and similar programs was found to exist.
In interviews with the three senior managers and departmental staff and managers (25 in all), 16 individuals specifically mentioned that to their knowledge there was no duplication of GBAP with any other programs. Only one staff member noted the potential of some level of internal duplication but for only 10 percent of the work.
The fundamental design elements which translate the principles of an ecosystem approach into a well-defined program structure are lacking (not adequately defined in documents both for Priority Ecosystems [PE] and for the Georgia Basin Action Plan [GBAP]).
There is evidence showing a role for government in this program area.
A valid role exists for the federal government, based on documents and interviews.
The public interest is served by the program; however, there is no apparent comprehensive and analytical examination of targeting reach, even though partnership constitutes one of the program principles. The unbundling process obfuscates the clarity and roles of a program such as the GBAP, an area-specific ecosystem initiative.
Theoretically there is a link associating this program with departmental outcomes from the OPP to the OPG, as well as Board outcomes and Board priorities.
Both documentation and staff interviews illustrate that there is a connection between this program and overall departmental strategic outcomes.
There is wide-ranging opinion on whether adjustments to the program are necessary to ensure better alignment with departmental priorities.
Due to a lack of relevant documentation, we cannot comment extensively on whether all OPPs within the scope of the OPG need to exist.
To answer the question of program duplication, EC staff state universally that no duplication exists. This evaluation did not specifically engage other methodologies to consider program duplication.
This section will examine success—perceived results of the GBAP, both intended and unintended, in areas supporting the mandate of the program.
| Q9 Evaluation questions | Statement of what should be observed | Rating |
|---|---|---|
What has happened as a result of the program? Have any outcomes been achieved as a result of the program? What have been its environmental impacts? |
The program demonstrates results in keeping with intended outcomes and planned deliverables. |
|
Given that documentation of outcomes and project deliverables was not readily available (i.e. no complete inventory of projects exists), comprehensive, or complete, little to no objective evidence of outcomes exists. We find some success in achieving outcomes at a project level, anecdotally.
No rigorous performance measurement data were supplied by the program; therefore, it is difficult to comment on attribution of program outputs to outcomes.
The GBAP has produced one report, the GBAP 2005 Update, which provides a brief overview of the deliverables of 16 projects (out of a total of 77); however, no link is provided between these projects and GBAP's outcomes outlined in its 2004 Logic Model. The Departmental Performance Report (DPR) also provides an account of activities in 2005-2006 dealing with, above all, partnerships, awareness, science and knowledge exchange, etc., but with no link to outcomes identified in the 2004 Logic Model.
Overall, almost all PTLs surveyed felt that the universe of immediate outcomes stated in the 2004 GBAP Logic Model was partially to fully achieved (Note: no answers were provided for "Enhanced awareness among communities of interest of environmental issues and the socio-economic impacts of their decisions). Regarding attribution to the GBAP program, the majority of PTLs surveyed felt that the universe of immediate outcomes stated in the 2004 GBAP Logic Model was partially to fully attributable.
Also, almost all PTLs surveyed felt that they gained moderate to substantial benefit from the GBAP in areas such as leveraging resources; forming partnerships with various stakeholders/players; accessing information, research and/or science; shared public profile on specific issues; and ability to inform senior management and decision makers on policy issues.
EC staff: Almost all felt that the GBAP was effective in leveraging resources (both cash and in-kind contributions). However, no details on such leveraging were available.
ENGO partner agencies: three comments provide mixed reactions to leveraging.
Focus Group (partnerships able to leverage resources): Comments mentioned issues surrounding transparency of funding criteria and process (3 of 7); overall, most comments critical of GBAP model and activities.
EC staff: GBAP has been successful in developing and improving partnerships with various key players/stakeholders, including provincial and local government, environmental non-governmental organizations (ENGOs), other federal partners, and business and industry. There was a mixed review regarding the development and improvement of partnerships with First Nations (capacity was cited as an issue on both sides); success with First Nations seemed to be a case-by-case issue.
ENGO partner agencies: Partnerships with various players have either been enhanced or are being established. Partnerships with First Nations groups are especially being pushed by EC (also ENGOs).
Focus group (partnerships in general): More than half of comments stated that the GBAP was not instrumental in forming partnerships (partnerships established under GBEI; issues with transparency in funding model); about a third of comments mentioned that the GBAP has been instrumental in forming partnerships, albeit as a silent funding partner (issues surrounding outreach, especially to industry and other federal departments).
EC staff: Data, for the most part, are available, although inconsistent (case-by-case scenario—Science and Technology (S&T) data reports seem to be available to those who need them in a timely manner, and these reports are integrated into the decision-making process). Some mentioned that the GBAP Website is not effective in providing the necessary information. Access to data itself does not seem to be a major issue (data do exist); the problem is that the information is not communicated properly to users; there is no information management policy; data are not being used effectively (integration into decision-making process).
ENGO partner agencies: Data/information has been shared to a certain extent—no real indication from the responses as to whether or not the data are integrated and support decision making (one comment out of three stated that the Planning and Implementation Teams (PITs), especially the Information Management Sharing PIT, have not worked).
Focus group (question on use of science and information): There seems to be a mixed reaction regarding the communication/dissemination (outreach) of information to users (one comment mentioned that a project was underway, yet no communication of the existence of that project or information from that project to users).
EC staff: Overall, communities of interest are engaged in the use of decision-making tools and joint planning, although not in a consistent or comprehensive fashion across the Basin. A few mentioned that data/tools are in early stage of development.
ENGO partner agencies: one pertinent comment stated that some PITs have used tools effectively, but joint planning has been a problem, especially where EC has been the lead.
EC staff: Strategies and plans, for the most part, incorporate sustainability principles, although not in a consistent or comprehensive fashion (e.g., Smart Growth, farming practices, GVRD). A few mentioned that on paper, this is the case; however, not necessarily with implementation of the plans.
ENGO partner agencies: Given that sustainability is a concept that is at the forefront of many levels of discussion, all responded in the affirmative, though two specifically mentioned that they already had such a concept in their plans and strategies.
EC staff: Most EC staff mentioned that either there had been actual acquisition of areas with ecological values or awareness was raised regarding sensitive areas. Again, this seems to be on a case-by-case basis, as a few mentioned that this was not relevant in their projects.
ENGO partner agencies: To some extent, though no attribution to the GBAP.
Focus group (question on changes in management of land use and protection of ecological values): Many specific examples provided (no mention either way of attribution to GBAP)–GVRD; Best Management Practices Code, etc.
EC staff: Most EC staff mentioned that there have been initiatives to prevent and reduce pollution, but again, this seems to be on a case-by-case basis, with no consistent or comprehensive strategy for the GBAP.
ENGO partner agencies: There seem to be some government initiatives to prevent and reduce pollution.
EC staff: Overall, more than half of EC staff mentioned that there seems to be some improvement; however, this seems to be on a case-by-case basis (air and water quality). Furthermore, measurement and attribution were mentioned as difficult processes, especially given the fact that a few mentioned that there were no benchmark indicators at the general level or follow up in place.
First Nations representatives: For the most part, First Nations representatives felt that the natural environment was not improving (in some cases, actually worsening)—not holistic (water quality, but not better sewage treatment, restrictive environment around ports).
Local government representatives: Mostly positive feedback, though specific examples were provided (air quality, invasive species, urban planning in specific regions—no discussion at ecosystemic level).
First Nations representatives: It was felt that scientific knowledge was the predominant source of data/information. Overall, reps felt that traditional knowledge (on a broader-based level) was not being used; one person mentioned that local knowledge seems to be used more as it is project specific.
Local government representatives: Scientific knowledge is used (scientific studies, new technology/tools, S&T reports). No mention of use of traditional ecological knowledge by any of the reps (one mentioned explicitly that they did not have a linkage with First Nations, specifically for land planning).
ENGO partner agencies: Seems to be project-specific; in some cases, scientific knowledge is being used; in others, they focus on traditional knowledge. One partner felt that the GBAP did not contribute to an increase in the use of traditional knowledge, as this is seen as a province-wide phenomenon; another discussed GBAP's direct and significant contribution to this (through the creation of a First Nations council, conferences), as it pertains to air and water quality.
First Nations representatives: Almost all representatives felt that there was a positive change in the level of responsibility being taken by communities, citizens, and industry. Two representatives attributed this partly to GBAP activities.
Local government representatives: Industry seems to be taking on more responsibility for its actions (however, there are increased regulations, codes of practice, etc.). There are problems achieving significant behaviour changes in individuals (media efforts are not effective enough).
ENGO partner agencies: There has been an increase in various players taking responsibility for their actions, but it does not seem that the GBAP contributed to this.
Focus group (question on taking responsibility): There was a mixed reaction in terms of attribution of various players taking responsibility for the activities of GBAP. GBAP's strength seems to lie in the fact that it does not play a direct programmatic role (playing a support/enabling role), but provides partners with scientific information and funding. The GBAP's weakness seems to be that it does not have a clear mandate/strategic direction.
First Nations representatives: Although partnerships are a good mechanism through which to raise awareness of First Nations environmental concerns, there were mixed comments regarding the level of various players coming together as partners. Furthermore, one mentioned the GBAP specifically as a venue to discuss their concerns, whereas another stated that it was the First Nations group that created partnerships in a specific area (the GBAP provided funding as a partner).
Local government representatives: More than half felt that there have been changes (due in part to resourcing/leveraging); one individual mentioned provincial politics as a barrier to partnerships (e.g., under-staffing, under-funding in the British Columbia Ministry of the Environment).
ENGO partner agencies: GBAP has contributed to various players coming together and created partnerships (though one comment out of four mentioned that this was a province-wide phenomenon and not limited to the Georgia Basin).
Population increases; the receptive community, the circumstances, the sense of urgency to motivate action, not just political will but also community will.
Capabilities or capacity of partners to participate: 12 of 19 respondents stated that this was from "somewhat of a challenge" to a "major challenge".
Level of stakeholder awareness and understanding: 10 of 19 respondents stated that this was at least "somewhat of a challenge".
Overall administration and management: 14 of 19 respondents stated that this was from "somewhat of a challenge" to a "major challenge".
Continuity of senior management commitment and engagement: 10 of 19 respondents stated that this was from "somewhat of a challenge" to a "major challenge".
| Q10 Evaluation questions | Statement of what should be observed | Rating |
|---|---|---|
What are the implications for Canada's economic growth and competitiveness? |
The program may have impacts on Canada's economic growth and competitiveness. |
|
Summary: There is a potential implication for Canada's economic growth and competitiveness; this linkage is at the early stages and no definitive attribution/conclusions can be drawn. The fact that GBAP's environmental results could not be readily demonstrated makes the further links to economic performance that much more tenuous.
In general, the Millennium Ecosystem Assessment states that ecosystems have contributed to substantial net gains in human well-being and economic development; communities depend on surrounding ecosystems for raw materials and essential services such as water supplies, food, timber and fuel. Economic systems also depend on the surrounding ecosystems for access to raw materials used in manufacturing, such as water, fuel and timber, among others (the Georgia Basin region supports such activities as agriculture, forestry, fishing, manufacturing, and recreation and tourism); therefore, the economic and public health costs associated with damage to natural ecosystem services can be significant.
Specifically, in the GBAP region, there are significant potential economic implications surrounding the tourism industry. There may be significant losses due to a decrease in air quality (leading to poor visibility). These losses have been predicted to be as high as $7.45 million and $1.32 million (for extreme visibility events) for the Greater Vancouver region and Frasier Valley, respectively (the study from which these figures stem also provides figures as low as $4.03 million and $.5 million for the two regions for lesser events). Unfortunately, sufficient data regarding the frequency of these events are not available; though the impact of a decrease in air quality would lead to losses in revenue from tourism, the likelihood of these events occurring is at this time not available. Therefore, though there may be potential negative economic implications stemming from a decrease in air quality in the region, the extent, in terms of revenue loss, of these negative implications cannot be adequately determined.
The GBAP action plan result does state that they will contribute to a strengthened economy; in that light, GBAP funding (Grants and Contributions) has promoted the development of economic growth; specifically, coastal management plans have been developed to diversify and expand the economies of many of B.C.'s coastal communities (focusing on such activities as shellfish aquaculture, log handling and storage, and public and commercial recreation). For example, the GBAP Strategic Outreach Project focuses on shellfish remediation initiatives which aim to re-open closed shellfish growing areas in order to harvest shellfish products for food, ceremonial and economic purposes.
| Q11 Evaluation questions | Statement of what should be observed | Rating |
|---|---|---|
Have there been any unanticipated results, either positive or negative, that can be attributed to the program? If so, how were they addressed? |
|
|
A wide range of unanticipated outcomes (nearly two dozen) were noted in interviews in the survey as well as in the one focus group. The majority of the unintended outcomes fell in the positive category; virtually none of the outcomes, either positive or negative, seemed to have been formally addressed by any of the program's management interventions.
Fifteen of twenty-two EC staff and managers spoke of unanticipated outcomes. On the positive side, they noted networking with other federal and provincial ministries; increased use of the Ecological Gifts Program where land donations yield tax breaks to property owners; addressing wildlife concerns for farm and dairy operations; increased public support for clean air initiatives; increased scrutiny by the public of the 2010 Olympics from a sustainability viewpoint; addressing digging in Boundary Bay; making important connections with the United States Environmental Protection Agency, new partnerships that are transferable outside of the limits of the Georgia Basin; as well as aiding the development and patent for a water sampler. On the negative side, mention was made of the difficulties in building trust with the First Nations and some staff reluctance to work with First Nations after hearing their complaints; the degree and extent of unhappiness of partners (DFO, partners, province); loss of trust and stability of some partnerships, the bureaucratic nature of the process which is reliant on old administration/control and contribution agreements, a shifting EC structure and attendant funding, inability to maintain launched GBAP projects throughout their lifetime, lack of tools to transfer the science, and inability to measure ecosystem progress. One also noted pollution elsewhere was more severe, yet the GBAP offered little solutions for those areas such as Prince George.
Three of five partner agencies noted items here; all were positive. The GBAP triggered discussions on environment both in the Coast Salish community as well as across the coalition of communities across the Basin. Some of the water lessons learned (water bucket and balance) have developed tools at BC grassroots that could be applied nationally. Projects that protect land (e.g., acquiring vulnerable Crown land and leasing it back to the BC Parks for requisite protection were also mentioned.
Only two of the seven First Nations interviewed offered positive outcome comments here. The GBAP seems to have unified their youth community on the subject of environment (which some contend was an intended outcome) as well as identified a pollution source for shellfish (namely, cattle).
Five of eight local government representatives offered comments all as positive outcomes. Reference was made to the best management practices for an agricultural study that showed which information gaps still need to be addressed as well as provided better farming practices for specific environmental concerns. The adoption of new technologies (e.g. Google Earth) provided an important opportunity to integrate and view data spatially. As well, two water projects were noted as advancing developments significantly, namely, the water balance model and Silver Ridge's storm water monitoring project.
The focus group noted improved facilitation and information-sharing among communities across the Basin, citing the Abbotsford auto recycling code now adopted across the province as a good example of a positive unintended outcome.
The survey showed that only 4 of the 17 PTLs noted unintended outcomes and all were positive. These were: additional projects identified; development of DNA-based bacterial source identification method; new scientific information; shared decision support criteria for prioritization of conservation acquisitions; common reporting template for priority acquisitions and the rationale for their selection; new liaison with groups; strengthened partnerships; and increased capacity through partnerships.
Given that documentation of outcomes and project deliverables was not readily available, comprehensive, or complete, little to no objective evidence of outcomes exists. We find some success in achieving outcomes at a project level, anecdotally.
Neither rigorous nor systematic performance measurement data were supplied by the program; therefore, it is difficult to comment on attribution of program outcomes to outputs.
There is a potential implication for Canada's economic growth and competitiveness associated with the programs; however, this linkage is at the early stages and no definitive attribution/conclusions can be drawn. The GBAP's environmental results could not be readily demonstrated, making the further links to economic performance that much more tenuous.
Broad and wide-ranging unanticipated outcomes have been identified as occurring from the program based on interview comments.
This section will examine cost-effectiveness—the potential for GBAP to be made more effective in the use of resources and the pursuit of its mandate.
| Q12 Evaluation questions | Statement of what should be observed | Rating |
|---|---|---|
Are there better ways of achieving the results? Have alternative programs been examined that might achieve the objectives and intended impacts and effects? |
Alternative delivery methods have been analyzed. |
|
The concept and general validity of an ecosystem approach exists in the literature, but there have been no specific reports connecting that notion to the situation on priority ecosystems at EC. No documentation of cost-comparative alternative approaches was found to have been conducted.
There was a range of comments from the 22 EC staff and managers interviewed on how to improve program coordination but no insightful comments on fundamentally different approaches. Some of the improvements suggested include improvements in coordination and outreach within EC; more frequent internal information and reporting, improving capacity to deal with partners both inside and outside EC; stronger leadership from senior management, and a comprehensive Website which documents all projects undertaken.
When asked about how they would improve outcomes the next time, 6 of the 17 respondents to the survey of Project Team Leads said, "More active pursuit of new partnerships with communities, the voluntary sector, and/or the private sector"; 5 noted "More efficient use of available resources in the delivery of the project(s)"; 4 pointed to "A less complex approach to the governance or structure (design) of the project(s)"; and 2 wanted "Better leveraging (use) of existing partnerships with communities, the voluntary sector, and/or the private sector".
| Q13 Evaluation questions | Statement of what should be observed | Rating |
|---|---|---|
Efficiency—If the program or activity continues, how could its efficiency be improved?8 |
Program or activity shows opportunity for efficiency increases. |
|
Salary and O&M account for almost 70 percent of total program expenditures per year; Grants and Contributions account for between a fifth to a quarter of total program expenditures.
The 22 EC staff and managers who gave an opinion on the program's efficiency (10 did not) were divided. Five felt there was an opportunity to improve efficiency whereas 7 did not. Some of the comments on improvements revolve around initial allocation of funding and opportunity for regrouping later, being willing to allocate resources to projects proposed by the partners and not only EC proposals, providing more time to secure greater buy-in from partners, much uncertainty on how to react to the unbundling process and the revised role of the GBAP Coordination Office in this advocating for unbundled project funding, and criticism of the operation of the Management Information System.
| Q14 Evaluation questions | Statement of what should be observed | Rating |
|---|---|---|
Should the program or activity include a cost recovery element? If yes, does it? |
Delivery of customized goods/services for defined client groups. A cost recovery mechanism is present; if applicable. |
n/a |
| Q15 Evaluation questions | Statement of what should be observed | Rating |
|---|---|---|
Value for money—Are Canadians getting value for their tax dollars? Is the program or activity cost-effective?9 |
Program demonstrates value for money. |
|
The financial data supplied and available shows only the planned allocations at the project level; no actuals were supplied. The overall GBAP budget has shown that the total amount of funding supplied has increasingly not been spent over the years (see budget tables below).
No reports or indicators showing attribution between program activities/outputs and outcomes other than at logic model level were found.
Salary and O&M account for almost 70 percent of total program expenditures per year and Grants and Contributions account for between a fifth to a quarter of total program expenditures. No data were available to indicate the types of activities to which O&M were applied.
No rigorous performance measurement underway to show project or program progress.
A management information system for tracking projects is not maintained or used for decision making.
O&M dollars were consistently under-spent compared with planned in all three years: by 10 percent in 2003-2004, 7 percent in 2004-2005 and 20 percent in 2005-2006.
The validation comment from the program manager states, "While the Salary and O&M indeed appear high, more extensive tracking and reporting of data at project levels would also help differentiate between clear overhead costs and the O&M used for contracts, MOUs and other direct project activities—giving a more realistic and less excessive-looking picture of the balance of expenditures. While this information can be found, the point that these were not easily accessible is of greater importance and I do not recommend we go back to unearth the specifics at this stage."
The validation comment from the program manager confirms that, "There is no systematic performance measurement data."
OVERALL—leads one to suspect that Canadians are not optimally getting value for money.
| Salary | EBP | O & M | Capital | G & C | Total | |
|---|---|---|---|---|---|---|
| Allocated | 1,390,500 | 278,100 | 1,799,100 | 66,000 | 967,300 | 4,501,000 |
| Actual | 1,360,066 | 272,013 | 1,400,991 | 66,000 | 937,931 | 4,037,001 |
| Difference | 30,434 | 6,087 | 398,109 | 0 | 29,369 | 463,999 |
| % Difference (Rounded) | 2.2% | 2.2% | 22.1% | 0% | 3.0% | 10.3% |
| Salary | EBP | O & M | Capital | G & C | Total | |
|---|---|---|---|---|---|---|
| Allocated | 1,561,196 | 312,239 | 1,999,734 | 0 | 1,036,329 | 4,909,498 |
| Actual | 1,434,995 | 286,999 | 1,778,793 | 0 | 1,045,075 | 4,545,862 |
| Difference | 126,201 | 25,240 | 220,941 | 0 | -8,746 | 363,636 |
| % Difference (Rounded) | 8.1% | 8.1% | 11.0% | 0% | -0.8% | 7.4% |
| Salary | EBP | O & M | Capital | G & C | Total | |
|---|---|---|---|---|---|---|
| Allocated | 1,667,977 | 333,595 | 2,406,982 | 160,000 | 810,125 | 5,378,679 |
| Actual | 1,434,233 | 286,847 | 1,640,435 | 103,025 | 818,226 | 4,282,766 |
| Difference | 233,744 | 46,749 | 766,547 | 56,975 | -8,101 | 1,095,914 |
| % Difference (Rounded) | 14% | 14% | 31.8% | 35.6% | -1.0% | 20.4% |
| Q16 Evaluation questions | Statement of what should be observed | Rating |
|---|---|---|
Affordability—Is the resultant package of programs and activities affordable? If not, what programs or activities would be abandoned?10 |
The program is financially affordable without the need to abandon components. |
|
Summary: It is difficult to state whether the program is affordable.
Cannot conclude whether or not the program is affordable; actual expenditure is lower than resources allocated for all three fiscal years. Yet without any comparison with similar programs, it is difficult to state whether or not the program is indeed affordable or whether the underlying rationale used to determine the level of resources allocated to the program each year is erroneous (is the funding allocated to the program on target or overestimated—if overestimated, to what degree is it overestimated?).
Of the 9 responses received, 5 felt that there was sufficient funding for the present GBAP activities. A few cited that given EC's changing structure and the unbundling of priority ecosystems, it is difficult to know where funding is coming from (source of funding). Of the 19 responses received in the PTL Survey, 10 stated that there would be a need for some change to activities or focus (3 stated it could continue with no changes, while no one stated that it could not continue at current funding levels).
It is difficult to state whether the program is affordable.
Alternative delivery approaches were not specifically researched as part of the evaluation methodology and documentation on this does not exist.
Given that efficiency of a program is a calculation based on the amount of outputs generated by dollar of resources input, and we could not make such a calculation, the question of whether GBAP is efficient cannot be answered. This is problematic. It was found that there was no consistent nor complete available listing of projects (and their outputs) for the GBAP. It was also found that the resources available were increasingly under-spent from FY2003-04 to FY2005-06. In the absence of performance reporting, success/effectiveness cannot be demonstrated. This brings cost-effectiveness/value for money into question.
Cost recovery is not an issue for this evaluation given that no specific clients receive custom benefits.
Given that no attribution between program activities/outputs and outcomes can be found, and in the absence of performance reporting, success/effectiveness cannot be demonstrated. Thus cost-effectiveness/value for money is brought into question, and the structure and ecosystem approach for PEs remains undocumented. All this leads one to conclude that there is no demonstrable evidence that Canadians are getting value for money.
This section will examine design and delivery—the clarity of activity, accountabilities, expected deliverables and intended results of the GBAP. In addition, process considerations pertaining to the allocation of resources, management of risk, monitoring and reporting, and the leveraging of partnerships were considered.
| Q17 Evaluation questions | Statement of what should be observed | Rating |
|---|---|---|
Does the program identify clear deliverables and expected results? |
Expected results and deliverables are clearly communicated and identified at program and project level. |
|
Expected results and deliverables are clearly identified in the 2004 GBAP Logic Model, though not all near-term results in the Logic Model are identified in the GBAP OPP (the near-term results for which the two Outcome Project Sub-components are responsible).
The majority of EC staff found the goals and objectives of the GBAP to be too broad and at too high a level to be useful operationally; all three signatory partners mentioned that the GBAP's goals and objectives are not clearly communicated to partners and were therefore not clearly and commonly understood by all partners. Additionally, most EC staff and signatory partners felt that most project level deliverables and milestones reflected the GBAP's goals and objectives.
Of the 19 responses received in the PTL survey, 15 stated that the goals and objectives of the GBAP were at least "somewhat clear or partially understood" (two stated "not clear or understood"; only 1 stated "very clear and well understood").
Some of these near-term results have been unbundled; however, the OPG does not provide a clear link between the specific accountabilities of other OPPs/OPGs (for which near-term results they are accountable) regarding PEs and how these accountabilities contribute to achieving the end results of the PE OPG.
Therefore, communication is an issue, certainly amongst the PTLs.
| Q18 Evaluation questions | Statement of what should be observed | Rating |
|---|---|---|
Are the activities and outputs of the program consistent with its mandate and plausibly linked to the outcomes in terms of clarity and attribution? |
|
|
The GBAP logic model draws links between the activities and outcomes; however, the model fails to indicate the development and application of a formal ecosystem approach. Such a methodology is not developed nor guides the actions/projects under the GBAP.
Of the 22 EC staff and managers interviewed, 12 provided comments on the linkage and even those comments were tangential to this issue.
The interviews with five partner representatives showed the two extremes, with one commenting that the GBAP never effectively addressed their concerns and another stating "it's a good return for money spent." The fifth responded (positive middle ground) by saying that the program grant permitted development of their water balance model and water bucket Websites.
For one of the program's client groups, industry, no contacts could be successfully identified for participation in interviews.
Data from the survey of the PTLs provided 19 observations. Answering the question on the degree to which the goals and objectives of GBAP were reflected in the project-level deliverables and milestones, 18 respondents said that these were "somewhat", "more" or "fully reflected" in their projects. Only one said it was weakly reflected and one said it was not reflected at all.
| Q19 Evaluation questions | Statement of what should be observed | Rating |
|---|---|---|
Are decision-making processes in place to allow for the highest areas of importance to be reflected in the allocation of resources (priorities)? |
Allocation of resources is based on highest importance and resourced according to priorities. | |
There is no apparent systematic decision-making approach to the selection of projects and resource allocation; no documentation was available. Budget allocation is addressed in another question.
This lack of project selection criteria was also specifically noted by one focus group participant.
| Q20 Evaluation questions | Statement of what should be observed | Rating |
|---|---|---|
How has risk been addressed? Has a risk management strategy been developed? Is it adequate? |
Risk is adequately addressed and managed. | |
The OPP for Priority Ecosystems identifies six areas of risk (four external and two internal):
Human and natural environment "Urban sprawl, increasing transportation demands and developments in the energy sector all result in very high risks to water, species and habitat, air quality and human health in the region. The risks of flooding, avalanches and landslides are increased through poor land use decisions".
Economic/trade/socio-political environment presents an opportunity—namely, "to build GBAP into the CESF bilateral where appropriate".
The legal framework presents an obligation and an opportunity. The GBAP is to meet its obligations with regard to these transboundary commitments with the US Environmental Protection Agency; as well, the GBAP is working on strengthening its partnership with Coast Salish First Nations.
Stakeholders - Formal signatory partners, as well as numerous local governments and non-government organizations, have made a commitment to participate in the GBAP projects and initiatives.
The departmental workforce is obliged "to meet the reporting requirements and to support the larger collaborative goals of the GBAP".
Business activities and assets present an "opportunity to focus and enhance communications and engagement with key stakeholders".
The Outcome Project Summary for the GBAP notes six specific risks and their management responses
Risk to the partnership and therefore to the commitment to GBAP outcomes is constant given other pressures facing partners. This risk is managed at the partnership level through the governance function by assisting partners in seeing the success of their contributions, be they funding or in kind. The opportunity to show on-the-ground successes shows the partners in a good light and assists in keeping them on board.
Flexibility in our business approaches such as an ecosystem or watershed approach helps put the focus on place-based improvements and assists in keeping the other partners engaged.
Economic / trade / socio-political environment risks are mitigated through clear communications within EC and with EC's partners, in particular the B.C. Ministry of Environment.
Risks related to legal frameworks, such as the current GBAP-Environmental Protection Agency agreement, are mitigated through the ongoing work and coordination within the GBCO and other EC units.
Risks related to changing First Nations standing, including recent court decisions, are mitigated through the ongoing development of relations with First Nations, notably through the full partnership status of Coast Salish in the GBAP. At the same time, risks to a balanced GBAP agenda and full Coast Salish participation require solid steering committee commitment and increased funding.
The GBAP is dependant on partners and stakeholders for its success; GBAP resources have previously facilitated these relationships. Current resource allocations are sufficient to maintain EC's status amongst partners and stakeholders and they continue to facilitate these relationships, with the possible exception of Coast Salish participation.
In interviews with the three senior managers, only one spoke of risk associated with the unbundling exercise. The GBAP becomes dependent upon a risk assessment from other outcome project groupings; this means other OPGs could eliminate a valid risk identified by GBAP. This, however, can be managed through effective partnerships at the Board and OPG level.
Interviews yielded specific comments from 20 program staff and managers; they spoke diversely of how risk was handled. One clearly said that there was no practice to identify and manage risks and it was but a paper exercise whereas another stated that risks and priorities are managed alongside the GBAP. Some other risks noted by others included: the risk of not doing ecosystem management properly, risks not been being collectively managed across project teams; and the risks of not handling partnerships properly. Another staff member voiced the need to manage very specific local risks such as the Vancouver Winter Olympics in 2010 or specific agricultural risks. Finally, another remarked on the risk that the GBAP put more energy into process than results.
In summary, there is formal program documentation of risks and opportunities, all of which are positively described. However, staff voiced a diversity and divergence of opinion; staff questioned whether risk was handled successfully.
| Q21 Evaluation questions | Statement of what should be observed | Rating |
|---|---|---|
Is there a clear and compelling analysis of capacity requirements? Are any proposals for increased capacity well justified? |
Presence and appropriateness of justifications for funding and resourcing. | |
In the Outcome Project Summary, capacity issues are raised. It states, "We lack the appropriate capacity to present our science and information in a context that will resonate with decision makers." It also notes that "Capacity to support First Nations inclusiveness remains one of continuing to build the relationship and is very much a funding issue."
The PTL survey revealed that of the 19 respondents, 12 felt that the degree of available resources (financial, people and material) more than partially met the needs of the project(s) as originally planned. Of the 7 who voiced a dissenting opinion, 5 said that this lack of capacity partially or greatly affected the achievement of the GBAP's goals and objectives.
There is no apparent documented analysis of financial capacity requirements.
There is no apparent need for additional financial capacity with much under-spending (2005-2006: Actual vs. Budgeted is ~ -22%).
No data exist on human resources capacity, although there seems to be a significant level of staff turnover in the GBCO.
| Q22 Evaluation questions | Statement of what should be observed | Rating |
|---|---|---|
Is there a clear link between program design, outcomes and the CESF pillars? |
Appropriate strategies are present in program design. | √ |
| Q23 Evaluation questions | Statement of what should be observed | Rating |
|---|---|---|
Partnership—What activities or programs should or could be transferred in whole or in part to the private/voluntary sector?11 |
Partnerships have been explicitly and comprehensively explored |
|
The ability to draw sufficient numbers of stakeholders was severely constrained. The program was not able to provide an accurate list with a sufficient number of accessible stakeholders and readily available contact information to allow the needed samples for all of the proposed methodologies to be developed. As an example, no industry partners initially forecast could be found; "... not one industry contact could be identified for an interview". The plan to conduct a partners survey with local government, stakeholders, and others initially forecast was deemed unviable due to the small number that could be located; the forecast 50 partners (based on an estimate from the program) turned into just 13. The surveys were converted into interviews. The 5 focus groups of 50 participants initially forecast were collapsed to only one with 10 attendees due to lack of identified participants. This shows a serious discrepancy between initially perceived and actual reach of the program.
Given that it was difficult to locate partners, there is an apparent issue with regards to reach for the program. This is especially problematic given that the program is based on partnerships.
Even internally there was an over-prediction of EC involvement by the program; initially well over a hundred names were suggested as PTLs; however, just 35 names were finally confirmed, and of those, only 19 leads responded to the survey. There was no readily available inventory of PTLs.
Of the 13 EC staff and managers that answered the pertinent question, the vast majority (10) were of the opinion that partnerships contributed to the program in a valued and meaningful way. Some mentioned that for them much was built on pre-existing partnerships; partners whose interest and commitment was principally tied to the funding aspect and the under capacity of partners to deliver on their commitments. Additionally, all the three senior managers felt partners contributed significantly from their vantage point.
The comments of the four signatory partners were evenly divided. Two indicated there were advantages but two questioned the value of the partnerships. One stressed that changes are recommended due to a fundamental flaw of GBAP "...if we want partners to recognize added benefit, all need to be resourced for it [the project]."
Ten of the 13 EC program staff and managers interviewed felt that more partners could be scoped in. Those who were not in favour were concerned about scope creep and the need to solidify the existing partners before expanding further and noted the need to provide funding to entice further environmental non-governmental organizations (ENGOs). The survey of Project Team Leads was nearly evenly divided on this question: 8 felt the greater opportunity for others to be involved, whereas 10 questioned the sufficiency of others to do more and undertake more outside of government.
Quoting from the consultant's interim report, "The ability to draw sufficient numbers of stakeholders was severely constrained. One of the key assumptions made, and reaffirmed during the orientation for this evaluation, was that of an accessible, accurate and sufficient number of stakeholders, and readily available contact information to develop the needed samples for all of the proposed methodologies. In practice though, numerous obstacles were encountered in attempting to develop cohorts of local government, non-government and industry respondents for interviews, survey samples and focus group participants. Furthermore, the management information system does not collect or report on any contact information for partners. Hence these cohorts must be developed by referral from PTLs, as they are the only owners of such information. Adding to this, responses to voicemails and emails were extremely slow. All combined, this resulted in a significant amount of time being spent in attempting to contact respondents and in seeking such referrals."
The initial forecasted contacts were estimates based on targets confirmed by program staff. Yet the program (GBCO or program leads) did not have required lists of such names. As well, there was no inventory of projects from which to readily draw partner contact information.
Throughout the process, a list of contacts and repeated attempts to secure contact were recorded by the consulting company.
After the launch of this evaluation, the number of anticipated versus actual contacts decreased dramatically. Two planned surveys of 50 had to be converted into interviews of just some 13 individuals.
A number of individuals contacted that were identified as potential contacts by the program responded that they were not sufficiently acquainted with the GBAP or unable/unwilling to provide input. Included in this list were the Federation of BC Naturalists and 6 EC employees from the Pacific and Yukon region.
Initially 10 telephone interviews were planned with industry; however, no contacts could be identified by the GBCO or PTLs.
One example of a letter of invitation sent to potential interviewees is provided in Annex 4.
The process by which contacts were to be sought was defined (Annex 4).
| Q24 Evaluation questions | Statement of what should be observed | Rating |
|---|---|---|
How consistent is the program with its own proposed approach (has the program been delivered as designed)? |
The program is consistent with and follows its defined approach/methodology. |
|
Comments on program design can be based only on the documentation available such as the logic model. This model appears thorough, yet it, along with the main priority ecosystem program, demonstrates a weakness in use of the nebulous term of an "ecosystem approach". In addition, this term is not specifically defined in the GBAP logic model, yet it is referenced as a second-level immediate outcome: "all plans use an ecosystem approach". There are no clear links in GBAP between this second-level immediate outcome to a first-level outcome nor the requisite supporting outputs and activities.
As evidenced in detail in an earlier question (#19), the program could not provide any documentation showing why the projects were selected; this is not to necessarily imply that these were inappropriate selections, but rather that the decision making process and criteria for these selections were not available.
All available documentation notes only the planning element but not the actual delivery of the products. There is no evidence showing the essential connections between what was planned and what was actually delivered. There are comments in interviews throughout about the delivery of projects such as, for example, "water bucket and water balance", but its connection to the project level is undocumented.
The management structure of a steering committee, management committee, project implementation teams and coordination office all have documentation of their activity and operations as planned. As well, one session of the Coast Salish First Nations Advisory Forum has been held; though its operations have been described as being annual.
Though EC managers and five partner agencies were interviewed and project implementation team leaders were surveyed, none of their commentary provided insights into whether the program was delivered as designed.
| Q25 Evaluation questions | Statement of what should be observed | Rating |
|---|---|---|
Are Environment Canada management and staff, and partners, supportive of the goals and objectives of the GBAP? |
Program staff and delivery partners are aligned, comfortable and supportive of program design and delivery. | √ |
The 9 of 10 EC staff and managers who commented on this section gave a resounding yes to the alignment question. The one who did not was not entirely sure of the GBAP's objectives. The three signatory partners who commented also said yes. Finally, the 19 PTLs who responded to this survey question all reported satisfaction at or above the "somewhat satisfied" category, with a dozen in the stronger "moderately satisfied" category.
In summary, nearly all the interviews with EC staff and management, signatory partners and PTLs reported that program staff and delivery partners are aligned, comfortable and supportive of program design and delivery.
| Q26 Evaluation questions | Statement of what should be observed | Rating |
|---|---|---|
Is there an established structure that provides for responsive management and logically supports the achievement of goals and objectives? |
Program management structures support program delivery. | |
The management structure of a steering committee, management committee, project implementation teams and coordination office all have documentation of their activities and operations as planned.
There were 22 EC staff and management interviews that provided some insights into dimensions of management support for the program. In terms of resource allocation, there was a dichotomy between whether the one static budgetary allocation from the program across the projects was good from a stability standpoint and whether greater flexibility for changing priorities should have been built in. There was also divergence of opinion on whether the funding was or was not adequate in terms of amount. Two people noted the lack of specific targets which negated the possibility of effective monitoring.
There was mixed reaction to whether coordination and support was adequate. The majority had little comment in this area but there were two extremes of opinion expressed. Some felt "we are trying to run before we crawl", with more partners brought onboard before solidifying the base of operations. Some noted that the GBCO was overwhelmed and under capacity, the program lacked stable leadership, administration used too much funding, and there was a lack of a project inventory, whereas others commented that there was no issue with program coordination. One person remarked on the confusion concerning the MIS and the lack of a simplified two-page guide to its operation. In this evaluation's discussions with the GBAP, the failing of this system has been fully recognized by the GBCO.
| Q27 Evaluation questions | Statement of what should be observed | Rating |
|---|---|---|
Is there comprehensive monitoring and reporting on performance that allows management and staff to carry out their responsibilities and demonstrate results? |
Performance measurement and reporting monitors and reports on program operations and outcomes. | |
A performance measurement framework was developed under contract but not put into operation.
Interviews with 22 departmental staff and managers substantiated the lack of a systematic and rigorous performance measurement system. Some of their commentary said the MIS is a great idea but awkward to use; there are data accuracy and reporting issues; there is "no consistent reporting in results oriented format"; and there is no requirement to "plan for results". They also noted that the project leads were reluctant to report to the MIS and did the bare minimum; one described the MIS as "a badly designed text data base". Further comments were made that pointed to the lack of accountability between resource allocation and reporting on results and the fact that the measurement of results was "a celebrated success rather than the real state of the environment". The GBAP overall was characterized by one respondent as operating with "...not much in the way of evaluation/assessment/request of what we have done with the money". In those interviews only a couple of staff were satisfied with the MIS.
In the survey of PTLs, 17 of 19 said there is sufficient useful information for managing and reporting on projects, and 15 said that the administrative requirements were minimal.
There were no annual reports to link program impacts with adequate attribution; the one annual report that was produced in 2005 reported on only a small sample of projects (16 out of 77), with no statement on the status of the remaining projects that were funded by the GBAP.
In summary, no rigorous performance measurement data were supplied by the program; therefore, it is also difficult to comment on attribution of program outputs to outcomes
| Q28 Evaluation questions | Statement of what should be observed | Rating |
|---|---|---|
Who is accountable for the program? Are the roles, responsibilities, and accountabilities of all groups involved clear, defined, and commonly understood? |
Roles, responsibilities and accountability are clearly defined in program management structure. | |
A defined governance and management structure has been documented for GBAP; there is a steering committee, management committee, project implementation teams and coordination office, all of which have documentation of their roles and responsibilities.
As well, EC and its partners for this project (Fisheries and Oceans Canada, Parks Canada, British Columbia Ministry of Water, Land and Air Protection, the British Columbia Ministry of Sustainable Management and First Nations and local government) all have their mandates and interests outlined under the GBAP framework for collaboration.
However clarity begins to dissipate due in part to the unbundling of the GBAP in 2005 as part of the priority ecosystem unbundling initiative. The one collective program has now been segmented across 10 OPGs and 22 OPPs.
Interviews with 22 EC staff and managers revealed widely differing opinions on the clarity of roles, responsibilities and accountability. The degree of clarity was sought at different departmental levels. Of those who had a definitive answer, three felt that it was well understood at a national level and six did not. At a regional level four said it was clear, three said it wasn't and three were mid-range. When asked as to whether this was clear for senior management, two said it was and four said it was not, while another said it was clear only internally. And, finally, with and amongst partners three respondents felt it was clear and four felt it was not clear.
Three of the four signatory partners commented on this aspect; two felt it was clear and one said definitely not.
The survey of 19 PTLs showed that 17 of the 19 found that roles were somewhat clear or partially clear or better. There were but two who questioned the clarity for them.
In summary, one can see the great dilution of clarity. The unbundling exercise certainly has made the participation and attendant roles and responsibilities more complex. Certainly, opinion is wide ranging and it is only the PTLs that perceive that for the most part, the roles and responsibilities of all groups involved in the delivery are clear and commonly understood by the parties involved.
Certainly given the requisite departmental OPP/OPG structure, there are documented deliverables and results for the program. However, basic information such as an inventory of projects and their outputs as well as PTLs is not readily available. Communication of such information to PTLs is also an issue.
Some important gaps in the logic of the program exist (e.g., development/application of Priority Ecosystem Approach). These gaps cloud plausible attribution of outcomes.
There is no apparent systematic decision-making approach to project selection and resource allocation.
There is documentation of a risk management strategy but the extent of actual management actions on this is questionable. Moreover, there is no formal vision to date, partners are elusive and the unbundling is weakening connectivity to projects.
There is no apparent documented analysis of financial capacity requirements. There is no apparent need for additional financial capacity (in 2005-2006, actual vs. budgeted left a 22 percent surplus of funds). No data exist on human resources capacity analysis, although there seems to be a significant level of staff turnover in the GBAP office.
Theoretically, there appears to be a linkage between the program and the Department's strategic positioning/former CESF pillars.
In terms of partnerships, which is one of the principles of the program, an extensive survey of GBAP partners was planned for some 50 partners of the program; however, the survey was cancelled as only 13 partners could be successfully located. The evaluation was initially planned to have 5 focus groups with a total of 50 participants, composed of co-deliverers familiar with the GBAP; however, only one focus group with 10 participants could be assembled. No industry partners were identified by EC. Given that it was difficult to locate partners, there is an apparent issue with regards to reach for the program. This is especially problematic given that the program is based on partnerships.
There is no apparent documentation of a complete program design upon which to base an assessment of the actual delivery and thus comment on the consistency of program design versus delivery.
In terms of EC management and staff being supportive of the program, there are wide ranging opinions among staff on this issue. Surprisingly, there were problems finding the actual PTLs for GBAP projects; only 19 were successfully surveyed.
There is a management structure defined and operating. However, there is no extensive systematic decision-making approach/structure to ensure active links of the GBAP to management that would provide responsive support to the program.
In terms of accountability, a management structure was evident whereby a Board lead exists (ES Board), the OPG lead exists, and the OPP lead exists. However, there is no apparent documentation of a program design upon which to base an assessment of the actual delivery. Opinion is wide ranging and only the PTLs perceive that for the most part, the roles and responsibilities of all groups involved in the delivery are clear and commonly understood.
This section deals with a question which was posed specifically by the Departmental Audit and Evaluation Committee when this evaluation was selected.
| Q30 Evaluation questions |
|---|
What lessons can be applied to the OPG (higher order question about approach)? |
The three senior managers responded in interviews concerning the commonalities and potential lessons learned that could be transferable from the GBAP. They stressed the uniqueness of each of the six geographically based departmental initiatives, citing the GBAP as being one of the best ones to integrate across the different environmental elements. Other commonalities mentioned include the "general feature of partnerships and collective needs being met through an integrated approach of partnerships" and the fact that "all work towards governance and shared objectives". It was pointed out that the other initiatives "can benefit ... in that they can learn from the strengths of the others".
There were two lessons-learned exercises done on the Department's ecosystem initiatives: a deck on May 9, 2006, and a summary table on March 31, 2006. In both, the lessons and actions were summarized. No further documentation on the state of learning and actions based on such lessons has been found.
In summary, one can only observe that the unification of lessons learned across ecosystem initiatives is challenging at best. The findings and observations that apply to the GBAP in this evaluation require close scrutiny for each of the other six initiatives as well as for the entire PE OPG. There is recognition of the commonality of lessons learned, but the extent of follow-up on that remains unknown.
Below is an overview of the findings in summary format according to the four main issues of relevance, success, cost-effectiveness and design and delivery.12
The fundamental design elements which translate the principles of an ecosystem approach into a well-defined program structure are lacking (not adequately defined in documents both for Priority Ecosystems [PE] and for the Georgia Basin Action Plan [GBAP]).
There is evidence showing a role for government in this program area.
A valid role exists for the federal government, based on documents and interviews.
The public interest is served by the program; however, there is no apparent comprehensive and analytical examination of targeting reach, even though partnership constitutes one of the program principles. The unbundling process obfuscates the clarity and roles of a program such as the GBAP, an area-specific ecosystem initiative.
Theoretically there is a link associating this program with departmental outcomes from the OPP to the OPG, as well as Board outcomes and Board priorities.
Both documentation and staff interviews illustrate that there is a connection between this program and overall departmental strategic outcomes.
There is wide-ranging opinion on whether adjustments to the program are necessary to ensure better alignment with departmental priorities.
Due to a lack of relevant documentation, we cannot comment extensively on whether all OPPs within the scope of the OPG need to exist.
To answer the question of program duplication, EC staff universally does state that no duplication exists. This evaluation did not specifically engage other methodologies to consider program duplication.
Given that documentation of outcomes and project deliverables was not readily available, comprehensive or complete, little to no objective evidence of outcomes exists. We find some success in achieving outcomes at a project level, anecdotally.
Neither rigorous nor systematic performance measurement data were supplied by the program; therefore, it is difficult to comment on attribution of program outcomes to outputs.
There is a potential implication for Canada's economic growth and competitiveness associated with the programs; however, this linkage is at the early stages and no definitive attribution/conclusions can be drawn. The GBAP's environmental results could not be readily demonstrated, making the further links to economic performance that much more tenuous.
Broad and wide-ranging unanticipated outcomes have been identified as occurring from the program based on interview comments.
Alternative delivery approaches were not specifically researched as part of the evaluation methodology and documentation on this does not exist.
Given that efficiency of a program is a calculation based on the amount of outputs generated by dollar of resources input, and we could not make such a calculation, the question of whether GBAP is efficient cannot be answered. This is problematic. It was found that there was no consistent or complete available listing of projects (and their outputs) for the GBAP. It was also found that the resources available were increasingly under-spent from FY2003-04 to FY2005-06. In the absence of performance reporting, success/effectiveness cannot be demonstrated. This brings cost-effectiveness/value for money into question.
Cost recovery is not an issue for this evaluation given that no specific clients receive custom benefits.
Given that no attribution between program activities/outputs and outcomes can be found, and in the absence of performance reporting, success/effectiveness cannot be demonstrated. Thus cost-effectiveness / value for money is brought into question, and the structure and ecosystem approach for PEs remain undocumented. All this leads one to conclude that there is no demonstrable evidence that Canadians are getting value for money.
It is difficult to state whether the program is affordable.
Certainly given the requisite departmental OPP/OPG structure, there are documented deliverables and results for the program. However, basic information such as an inventory of projects and their outputs as well as PTLs is not readily available. Communication of such information to PTLs is also an issue.
Some important gaps in the logic of the program exist (e.g., development/application of Priority Ecosystem Approach). These gaps cloud plausible attribution of outcomes.
There is no apparent systematic decision-making approach to project selection and resource allocation.
There is documentation of a risk management strategy but the extent of actual management actions on this is questionable. Moreover, there is no formal vision to date, partners are elusive and the unbundling is weakening connectivity to projects.
There is no apparent documented analysis of financial capacity requirements. There is no apparent need for additional financial capacity (in 2005-2006, actual vs. budgeted left a 22 percent surplus of funds). No data exist on human resources capacity analysis, although there seems to be a significant level of staff turnover in the GBAP office.
Theoretically, there appears to be a linkage between the program and the Department's strategic positioning/former CESF pillars.
In terms of partnerships, which is one of the principles of the program, an extensive survey of GBAP partners was planned for some 50 partners of the program; however, the survey was cancelled as only 13 partners could be successfully located. The evaluation was initially planned to have 5 focus groups with a total of 50 participants, composed of co-deliverers familiar with the GBAP; however, only one focus group with 10 participants could be assembled. No industry partners were identified by EC. Given that it was difficult to locate partners, there is an apparent issue with regards to reach for the program. This is especially problematic given that the program is based on partnerships.
There is no apparent documentation of a complete program design upon which to base an assessment of the actual delivery and thus comment on the consistency of program design versus delivery.
In terms of EC management and staff being supportive of the program, there are wide ranging opinions among staff on this issue. Surprisingly, there were problems finding the actual PTLs for GBAP projects; only 19 were successfully surveyed.
There is a management structure defined and operating. However, there is no extensive systematic decision-making approach/structure to ensure active links of the GBAP to management that would provide responsive support to the program.
In terms of accountability, a management structure was evident whereby a Board lead exists (ES Board), the OPG lead exists, and the OPP lead exists. However, there is no apparent documentation of a program design upon which to base an assessment of the actual delivery. Opinion is wide ranging and only the PTLs perceive that for the most part, the roles and responsibilities of all groups involved in the delivery are clear and commonly understood.
3 NOTE: Rating is a judgement on whether the findings indicate no major problem (√) or a small
problem (
) or a major problem (
). Annex 3 presents a summary list of such ratings
10 Expenditure review question
11 Expenditure review question
12 For a summary of ratings by question, see Annex 3
