Home > Proactive Disclosure > Audits and Evaluations > A&E Reports > 2008-2009 > Evaluation of the Meteorological Service of Canada (MSC) Transition Project

EXECUTIVE SUMMARY

In 2003, the Government of Canada announced increased funding to revitalize and transform the operations of the Meteorological Service of Canada (MSC). The objectives of this project, known as the Transition Project, were to help the MSC become a more sustainable organization, to advance science- and weather-related services, and to help Canadians adapt in ways that safeguard security, health and safety, economic prosperity and environmental quality. The government allocated approximately $74 million (M) in new funds over a five-year period and $5M per year thereafter. This funding was to be combined with $11M from internal allocations over the same five-year period and approximately $2M annually thereafter.

An evaluation of the progress made to modernize Environment Canada’s weather services over the 2003–2004 to 2007–2008 period was required by Treasury Board. The importance of evaluating this significant and transformative measure was also endorsed by Environment Canada’s Audit and Evaluation Committee¹ and the conduct of an evaluation was included in the department’s 2007–2008 Audit and Evaluation Plan.

The purpose of the evaluation is to assess:

• relevance: whether the Transition Project goals were consistent with federal priorities and addressed actual needs;
• success: the extent to which the Transition Project has met its intended outcomes;
• cost-effectiveness / alternatives: whether the most appropriate and cost-effective means were used to achieve those outcomes; and
• design and delivery: whether the Transition Project was delivered in the best possible way.

The evaluation addresses activities funded under the Transition Project over the five‑year project period from fiscal year 2003–2004 through 2007–2008. Multiple methodologies were used to collect information. These methods included

• document and file reviews;
• secondary data analyses and reports;
• a media scan;
• key informant interviews (58 people in total: 36 internal staff and 22 external stakeholders); and
• three facilitated workshops with Environment Canada employees (47 people in total).

The major conclusions of the evaluation are presented by evaluation issue.

Relevance

• The objectives and intended outcomes of the MSC Transition Project a) were aligned with Environment Canada’s mandate for meteorological services and departmental strategic outcomes as set out in corporate documents; b) were related to priorities identified in the Government of Canada’s speeches from the Throne and budgets; and c) addressed actual needs to modernize Environment Canada’s weather services.

Success

• The Transition Project has made progress on most of the activities undertaken. These activities have resulted in the consolidation of weather offices, a focus on high‑impact weather, increased training and recruitment of staff, the introduction of new products and services and the life‑cycle management and modernization of some equipment. Activities that have not been completely implemented and require further attention include the decommissioning and cleanup of some climate and hydrometric sites and the development and implementation of quality assurance processes and data management frameworks.
• Despite the achievement of these activities, the intended outcomes at the immediate and ultimate outcome levels will only be partially achieved by project end (2007–2008). For example, while there is evidence of ongoing recruitment in the Meteorology Group and Engineering and Support Staff categories, there is a widespread perception, both internal and external, that current succession planning efforts are inadequate to address the impact of an aging workforce. Notwithstanding the progress made to modernize monitoring networks through network rationalization, considerable work remains to be done. Steps have been taken to enhance the quality assurance of and access to key data, but the data management framework is still under development and quality assurance processes are considered to be under-resourced. Further, even though progress has been made in decommissioning sites and stations and installing new equipment at reference climate stations, the decommissioning of sites has not occurred at the rate expected. These partial achievements weaken the ability of the Transition Project to attain its overarching outcome, which is to make MSC a more sustainable organization.
• The progress achieved under the Transition Project was affected by factors internal to the project itself:
  1. The costs of some activities (e.g. consolidation and modernization of forecast offices) were underestimated and resources had to be redirected from other activities (e.g. from restoring and developing key skill sets).
  2. Staffing, training and consequently relocation processes took longer to get into place than expected (e.g. National Service Offices).
• The ability to implement activities and achieve the intended outcomes was also affected by factors external to the project:
  1. The ability to achieve outcomes was affected by department-wide changes to governance and resource allocation processes. Budget decisions became the responsibility of boards, not line-based organizations, and resources were centralized and allocated on a priority basis.
  2. Government decisions were not matched with funds (e.g. the cost of the offices in Gander, Newfoundland and Winnipeg, Manitoba), meaning resources had to be redirected from Environment Canada’s A-Base programs.
  3. The implementation of activities was delayed by policy requirements (e.g. land claims negotiations) and new procedures for the management of property.

Cost-effectiveness / alternatives

• Environment Canada interviewees widely agreed that the Transition Project used the most appropriate and effective means of achieving the intended outcomes and that there were no obvious alternatives. Although there is compelling qualitative evidence that the project provided value for the money received, a complete assessment could not be done due to a lack of financial information.

Design and delivery

• While the concept and design of the project was considered to be the best way of achieving the intended outcomes, the project was not delivered entirely as planned. Some activities were dropped and others have not been completed to date, notably in the monitoring component of the Transition Project.
• As previously indicated, MSC underestimated some of the funding requirements and resources were reallocated within the project.
• Pockets of dissatisfaction with the project exist among Environment Canada interviewees and some stakeholders. Environment Canada employees are concerned about the potential to achieve the outcomes sought and some stakeholders express concern over the delivery and quality of services and the adequacy of Environment Canada’s efforts to address the impact of an aging workforce.
• As a result of the inability to access all financial information, the evaluation was not able to confirm whether or not the project had remained within budget.

Recommendations

The following recommendations are based on the findings and conclusions of the evaluation. The first three recommendations are directed to the Weather and Environmental Services (WES) Board and the fourth recommendation is addressed both to the WES Board and the Departmental Management Services (DMS) Board.

1. Take stock and communicate which activities need to be continued in support of the modernization and sustainability of Environment Canada’s weather services.
   
   Given the ongoing importance of modernizing Environment Canada’s weather services, attention is needed to maintain the progress and momentum achieved to date. Priority attention, in particular, is required to ensure the sustainability of Environment Canada’s weather services. In deciding which of the activities initiated under the Transition Project need to continue, the WES Board needs to
   
   
   1. Identify the status of the activities and how the activities relate to the ongoing modernization and sustainability of MSC
      
      The overarching objective of the Transition Project was to help MSC become a more sustainable organization and infrastructure. The achievement of the other ultimate outcomes — strengthened linkages between production, science and service; improved services for Canadians and key stakeholders; and adaptation of Canadians to the environment in ways that safeguard security, health, the economy and quality of the environment — depend on the existence of a sustainable organization and infrastructure. Thus, in considering which activities should be continued, the WES Board needs to assess how these activities relate to the sustainability sought under the project and, further, how these activities contribute to the overall sustainability of Environment Canada’s weather services funded by existing A-Base resources.
      
      In this context, the WES Board also needs to consider the risk to the mandates and priorities of internal stakeholders (e.g. the Environmental Protection Board) and external stakeholders, including other federal departments and agencies, other levels of government, the private sector and Canadians at large of not continuing the activities launched under the "Transition Project" to align with what was sent for response.
   
   
   
   2. Determine the resource requirements of ongoing activities
      
      The modernization efforts of the Transition Project were to be complemented by other A-Base programs; hence resource allocation should be considered within the broader context of A-Base funding for weather services. Seven million dollars annually has been allocated for the implementation of activities initiated under the Transition Project. In considering the budget allocations, the WES Board needs to consider how to best use and leverage the existing resources (financial and human) provided under the project with MSC’s other A-Base programs.
   
   
   
   3. Determine the adequacy of current succession planning efforts
      
      External stakeholders in particular voiced concern over the impact of an aging workforce on the capacity of Environment Canada to deliver high-quality weather services. The WES Board needs to assess the capacity of Environment Canada and the adequacy of current succession planning efforts to support sustainable and high-quality weather services to stakeholders.
   
   
   
   4. Articulate a strategic direction and implement an action plan
      
      Based on the information collected above, the WES Board needs to decide which activities will continue to be funded, how the resources will be allocated and linked to existing A-Base programs, and what needs to be done to ensure the capacity needed to deliver quality weather services. The strategic direction should be accompanied by an action plan which would set out specific measurable deliverables, timelines and accountabilities.


2. Communicate the strategic direction and action plan
   
   The strategic direction and the action plan articulated by the WES Board need to be communicated to
   
   
   1. MSC staff and other internal stakeholders
      
      The vision for the Transition Project was clear and, overall, MSC staff were motivated to achieve the intended outcomes. Over time, Environment Canada interviewees² became discouraged by their perceived inability to achieve the intended outcomes. Modernization of Environment Canada’s weather services is an ongoing requirement of the department, and employees need to be clear about the direction of programming. The strategic direction and the action plan formulated by the WES Board need to be communicated to the MSC, to the Science and Technology (S&T) Branch and to the staff of the Chief Information Officer (CIOB) Branch.
      
      
   2. External stakeholders outside of Environment Canada
      
      Many external stakeholders are of the view that the 2004–2005 changes in the organizational structure of MSC and the turnover in personnel have moved Environment Canada away from being a sustainable organization. External stakeholders are concerned that Environment Canada does not have the capacity or succession planning efforts in place needed to create a stable organization. The changes in governance at Environment Canada, along with MSC’s efforts to address succession planning issues, need to be addressed with external stakeholders to ensure their continued support and the continued credibility of Environment Canada’s weather services.
      
      
3. Strengthen performance measurement, monitoring and reporting at the outcome level
   
   While information on the progress of the project was collected at the activity level, information on how these activities contributed to the achievement of immediate and ultimate outcomes was not consistently collected and aggregated. Despite the existence of a performance measurement strategy, that strategy was not implemented and information at the outcome level was not readily available. The absence of performance information weakened the ability of the department to tell its full performance story as to how weather services have been modernized over the past five years and to identify ongoing requirements for programming and resources. To demonstrate good stewardship and accountability for the use of public funds and informed decision making, the WES Board needs to strengthen performance measurement, monitoring and reporting at the outcome level.



4. Ensure that mechanisms to track and report financial information are in place
   
   The financial information system did not specifically track the use of Treasury Board Secretariat funds from the use of other A-Base funds or indicate the impact of government and department-wide budget reductions on the Transition Project. Financial information also needs to be linked to results to enable Environment Canada to determine and report on the cost-effectiveness or value for money of programming. To ensure accountability of funds and to demonstrate good stewardship over the use of funds, the WES Board, in conjunction with Departmental Management Services (DMS) Board, needs to clarify who is responsible for tracking the allocations, budget cuts and expenditures of new and A-Base funds (e.g. program managers or Corporate Finance) and what mechanisms will be used.

Management Response

It must be noted that the Meteorological Service of Canada is a much different organization now than it was in 2003.  Specifically, atmospheric and climate research is now conducted under the auspices of the Science and Technology Branch.  Similarly, information technology activities are carried out under the Chief Information Officer Branch (CIOB).  While Weather and Environmental Services Board strategically manages the work of MSC and Science and Technology (S&T) related to weather and environmental services, the Information Technology (IT) infrastructure critical to everything WES Board does is under the purview of the Departmental Management Services Board.  Therefore it will be critical to ensure that all recommendations are implemented in an integrated and horizontal fashion across organizations and Boards to ensure results.

WES Board agrees with the recommendation.  The ultimate outcome of a more sustainable organization and infrastructure with strengthened linkages between science, production and service remains a key goal. Significant work was done in first considering alternate service delivery models and ultimately in determining resource requirements to deliver the mandated responsibilities of a sustainable weather service.  This was the basis for the MSC Transition.  As the A-Base on which Transition was predicated has steadily eroded and as other internal and external drivers have evolved, it is timely to re-evaluate the path forward. 

There are a number of “stock-taking” initiatives that are already underway that are larger in scope than just the Transition activities and that will necessarily include them.  Given that CIOB is critical to the delivery of Weather and Environmental Services (WES ), it will be imperative that DMS Board is engaged in these WES Board activities:

• WES Board will contribute to an Environment Canada Strategic Review by validating its mandate, conducting a full review of the costs associated with its programs, identifying lowest priorities and associated risks by September 2008;
• Concurrently, WES Board will define a long-term strategic way forward that is aligned with the directions of society and government;
• Both of these “review initiatives” will guide recommendations for adjustments to core programs and services including the Transition activities in the form of a business case by fall/winter 2008/2009 designed to ensure continued relevance in a sustainable fashion; and
• MSC People-planning will underpin everything to ensure the right level of skill-sets are available to the MSC/WES Board to be a continuous learning organization into the future and to put in place the succession plans to get us there.

The following sections respond to the more detailed sub-recommendations by expanding on these themes and identifying timelines.

Management Response to Recommendation 1a)

WES Board agrees with the recommendation. The present Transition evaluation report goes a long way to identifying the status of the Transition activities. An Office of the Auditor General (OAG) evaluation of the MSC’s warning program is also expected to provide valuable insight into the continued relevance of those activities in contributing to ongoing sustainability (the first draft report is expected in summer 2008). These reports will inform the review initiatives and will result in recommendations as to which Transition activities need to continue. This will lead to the development of a business case in fall/winter 2008/2009.

Management response to Recommendation 1b)

The review initiatives will result in an understanding of the relevance and resource needs down to the program level and position us to recommend the path forward to meet our stakeholders’ and partners’ ongoing and increasing demands for environmental information and services.  As above, completion is anticipated by fall/winter 2008/2009.

Management Response to Recommendation 1c)

As a knowledge-based organization, MSC is critically dependent on its people.  The MSC has developed a People Plan to address succession planning and much more.  The plan comprehensively addresses the needs of the meteorological enterprise of Environment Canada to effectively integrate business and human resource planning to facilitate the attraction, development and retention of a highly effective and innovative workforce.  Again, integration with similar planning at Science and Technology Branch (S&TB) and CIOB will be critical in ensuring that the WES Board is fully equipped with the skill-sets required to move forward on its strategic direction and achieve the desired results.

The MSC People Plan is based on a departmental framework built around the People Component of the Management Accountability Framework (PCMAF) developed for the public service at large.  The plan is built around three key elements: Inclusive Culture, Building People Capacity and the Healthy & Respectful Workplace. Under these elements, there were 8 key areas identified and MSC Management Champions have been assigned to each one.  The Management Champions have established working groups and have begun working on implementation of the tasks identified in the plan.

For succession planning in particular the MSC has already identified critical positions, instituted a coaching and mentoring program for employees interested in moving into management and instituted learning Plans for 95% of its employees. While the MSC is making progress in succession planning, there are challenges with a potential bow wave of retirements as well as the need to maintain unique, national and international subject matter experts.  It is recognized that mentoring, coaching, job shadowing and developmental assignments are excellent mechanisms to mitigate this challenge, but scarce resources are impeding the MSC’s ability to move ahead as fast as it would like.

Financial resources needed to implement this people plan will be a part of the business case being developed.  MSC will also review the succession action plan to establish a mechanism to better communicate our approach to stakeholders.

Management response to Recommendation 1d)

The current review initiatives as described in this response so far are focused on delivering just this: a recommendation to government for a strategic direction and implementation plan by fall/winter 2008/2009.

Management response to Recommendation 2a)

WES Board agrees with Recommendation 2a).

Communications has always been a priority for WES /MSC .  Recently the MSC Forum was created as a mechanism for managers to exchange information, and communicate the regional perspective.  The MSC Forum is chaired by the Assistant Deputy Minister (ADM ) and includes Director Generals, Directors, Regional Directors and Planners.  The forum meets every six weeks and ensures that all aspects of MSC /WES Board operations are considered at the management table. This has helped to ensure consistent messaging from management to staff throughout the MSC.

The ADM MSC (in some sites with the ADM S&T Branch) is conducting cross-country Town Hall meetings with staff in the spring 2008.  The purpose of these meetings is to engage MSC, as well as S&T and CIOB staff critical to WES activities in a dialogue for the future.  These meetings will focus on successes, on WES strategic directions and on people planning issues and will be conducted on an annual basis going forward. 

Throughout the MSC Transition, employee surveys were conducted to assess the staff’s perceptions and identify concerns.  We have recently completed the 2007 MSC Employee Survey and will be reviewing the results in the next few months as well as the next steps. In addition, we will be launching an Intranet site as approved by MSC Management to provide enhanced information such as bi-weekly ADM news items, progress on our strategic exercise, updates on our people plan and a connection with WES activities.  A generic email has also been created for all staff to provide their comments and views on the future directions of WES. (WESVisionSME@ec.gc.ca)

The ADM MSC will continue to consult and communicate horizontally within Environment Canada through his Priority Management Board membership and participation.

Management response to Recommendation 2b)

WES Board agrees with Recommendation 2b). WES Board recognizes that the stakeholders’ concerns raised in this evaluation will need to be further explored and addressed.

We will continue to build on long-standing relationships with Other Government Departments, non-federal partners in academe, Emergency Measures Organizations and the provinces in communicating our strategic directions and action plans.  The MSC National Advisory Board – key to the development of the previous MSC Transition strategy – will be re-invigorated as a forum to seek input. Board membership is being updated to include a broad range of key stakeholders. The first meeting of the new Advisory Board will be held in the fall 2008 and the focus will be the strategic review initiatives. 

A proactive strategy is also being developed for the ADM MSC (WES Board delegate) to connect with other key stakeholders both internal and external to government on the evolution and direction-setting of WES programs. WES Board commits to the development and initiation of this strategy this spring.

Management response to Recommendation 3

WES Board and DMS Board agree with the recommendation. Significant work has been carried out over the past several years in identifying a variety of performance indicators for the WES program. More recently, the WES Board Quality Management System (QMS) has been working towards registration to the ISO 9001:2000 International Standard. This standard focuses on using performance information to continuously improve the quality of our information and services to better meet the needs of clients. 

We have completed Phase I of the QMS project which saw sixteen activities (Outcome Projects), including four CIOB activities, successfully ISO registered in 2007 and we are moving forward to expanding the registration to include an additional twenty one activities by the fall of 2008, essentially registering all WES Board operations.  ISO registration will certainly contribute to meeting this recommendation.

The QMS work will be entirely in step with the performance framework being developed in support of the Program Activity Architecture (PAA). This performance framework will identify metrics relative to all PAA levels right up to the ultimate Outcomes.  The QMS will stay linked to this effort to ensure consistency with the PAA objectives and metrics.  Together, these initiatives will produce an upward cascade of metrics showing the contribution of activities to intermediate results and ultimate outcomes.

WES Board commits to implementing phase two of the ISO 9001:2000 registration by fall 2008.  WES Board commits to implementing a new PAA with performance measures up to the ultimate Outcome level by fall 2009. Through these activities and the lessons learned from the MSC Transition evaluation, MSC will be well-poised to propose an effective performance measurement framework for future endeavours.

Management response to Recommendation 4

WES Board and DMS Board agree with the recommendation. 
 
For the first three years of the project the project funds including carry-forward were properly tracked.  A disconnect happened in the 4th year where the carry-forward was not tracked and budgeting was based on previous year’s expenditures. 

Proposed Implementation Plan

Following is an outline of the implementation plan for the Management Response to the Recommendations contained in the Evaluation of the MSC Transition Project.  Financial resources required for this implementation are not addressed herein.  In many cases, activities will fall under the responsibility of WES Outcome Project Group (OPG) 2B3 and as such will depend on adequate resource availability.

¹ In 2007, the Departmental Audit and Evaluation Committee was split into two components. The component that deals with evaluation is now known as the Departmental Evaluation Committee.

² The term “Environment Canada interviewees” refers to MSC staff and former MSC staff (now in the Science and Technology Branch).