Home > Proactive Disclosure > Audits and Evaluations > A&E Reports > 2008-2009 > Evaluation of the Meteorological Service of Canada (MSC) Transition Project

4.0 FINDINGS

The findings presented in this section address the four evaluation issues and the evaluation questions corresponding with each issue (see Annex B for the list of evaluation questions grouped by issue). The overall finding for each evaluation issue is presented first, followed by the findings for the corresponding evaluation questions.  Findings for evaluation questions three and five, which address the impact of changes in governance and external factors, are incorporated and discussed throughout the findings for the other evaluation questions.

4.1 Relevance

Overall Finding

• Yes, the objectives and intended outcomes of the MSC Transition Project were relevant to the mandate and priorities of the department and the Government of Canada.

EQ 1 Findings

• Yes, the objectives and intended outcomes were aligned with Environment Canada’s mandate for meteorological services and departmental priorities.
  • The document review demonstrated alignment with the Weather and Environmental Prediction Business Line 2004–2005 strategic outcome Help Canadians adapt to their environment in ways that safeguard their health, safety and security, optimize economic activity, and enhance environmental quality and, now under the WES Board, with the 2007–2008 strategic outcome Weather and environmental predictions and services reduce risks and contribute to the well-being of Canadians.
• Several topics in the Federal Budgets reviewed relate to topics associated with the MSC Transition Project.²⁶ In addition, topics from recent Speech from the Throne documents also related to topics associated with the MSC Transition Project.²⁷

EQ 2 Findings

• EC interviewees widely agree that the objective and five components of the Transition Project were needed and the right areas to focus on
  • Component 1:  Consolidation and Modernization of the MSC’s Forecast Operations: Interviewees note that the reduction of forecast centres was relevant and that it was necessary to demonstrate the efficient use of funds.
  • Component 2:  Creation of National Service Offices and Outreach Capacity: Interviewees are of the view that it was important for MSC to move forward with a client/service focus and that the creation of National Service Offices and outreach capacity was an appropriate step.  Interviewees also note that in some cases the geographic location of National Service Offices was influenced by the government.
  • Component 3:  Restoration and Development of Key Skill Sets:  It was noted during the interviews that recruitment of new employees was important as there was a period in the 1990s when there was virtually no recruitment and also that training and recruitment was particularly important within the context of the retirement of meteorologists and workforce demographics. Human resources documentation reviewed containing retirement eligibility and demographic-related data from the early part of the Transition Project period also reinforced the need for the restoration and development of key skills.
  • Component 4:  Product and Service Enhancements and Innovation:  The importance of this in light of changing technology was noted during interviews. An interviewee noted that in the past a lot of product development was done “off the edge of somebody’s desk.”
  • Component 5:  Invigorating Monitoring Capacity:  Environment Canada interviewees indicated that tackling the issue of rust-out was one of the biggest problems and part of the rationale for the project.  Interviewees also noted the importance of closing some sites and of moving to a life-cycle management approach to maintain the integrity of equipment.
• In the final report from the workshops conducted as part of the evaluation, it is stated that “[g]enerally, participants agreed with the goals of the Transition project and were supportive of the expected outcomes.  They appeared to value particularly the fact that the Transition project was an initiative, with funding, designed to change the MSC into a sustainable organization, within the constraints that it was facing in 2003.”²⁸
• While Environment Canada interviewees are of the view that the concepts and theory of the project were good, the project was not necessarily implemented as intended.  Environment Canada interviewees widely agree that the change of governance and budget allocation processes in 2005 ”derailed” the Transition Project from completing the implementation of these components and the achievement of its overarching objective.

4.2 Success

This section examines both intended outcomes (at the immediate and ultimate levels) and unintended outcomes of the project. Unintended outcomes are defined as outcomes that occurred but were not anticipated at the outset of the project.

Overall Findings

• The Transition Project has made progress towards but has not completely achieved all of its intended outcomes at the immediate and long-term (ultimate) level.
• Pockets of dissatisfaction are noted among Environment Canada interviewees and external stakeholders.
• The ability to achieve intended outcomes is affected by organizational and departmental changes in governance and resource allocation processes, unfunded government commitments, budget reductions, new policy requirements related to the management of property, unforeseen land claims issues, and an under-estimation of the time to set up the new National Service Offices and laboratories.
• Changes in departmental governance and resource allocation processes were considered by many Environment Canada interviewees to delay or weaken the implementation of the Transition Project and to fragment the services provided to outside stakeholders, contributing to the perception that MSC was becoming an unstable rather than a more sustainable organization.

EQ 4A Findings – Immediate Outcomes

Rating: Progress made, attention needed

1 a) Intended Immediate Outcome: Increased Efficiency of Forecast Operations

Consolidation of Forecast Operations

• Document review and interviews show that weather forecast operations were consolidated.  Seven weather centres were closed and five Storm Prediction Centres located in Dartmouth, Montreal, Toronto, Edmonton and Vancouver were established.  One additional Storm Prediction Centre was introduced in Winnipeg as a satellite office of the Edmonton centre. Two aviation centres were consolidated into the Edmonton and the Montreal Storm Prediction Centres. Five national laboratories with specialized focus points were collocated with the Storm Prediction Centres.  The focuses of the laboratories were marine and coastal weather (Dartmouth), severe weather forecasting (Montreal), Nowcasting (Toronto), arctic weather and hydrometeorology (Edmonton), and coastal and mountain meteorology (Vancouver).
• Some internal interviewees are of the view that the consolidation of forecast operations improved efficiency by reducing the number and thus operating costs of forecast offices.  Secondary data shows a reduction of over $8M in the budget allocations for weather forecast offices.²⁹ While this data shows reductions in budget allocations to the production component of MSC, this reduction needs to be considered within the broader requirements and expenditures of consolidating and modernizing the offices of the Transition Project itself.

Modernization of MSC’s Forecast Operations

• Interviews, secondary data and document reviews indicate that MSC’s Transition Project provided additional resources needed to implement an automated public weather forecast tool called SCRIBE. Secondary data and reports indicate a steady improvement in the accuracy of public forecasts for the five-day period.
  • Over the past two years, the SCRIBE team has also worked on developing a marine SCRIBE.
• Interviewees and the document review indicate that resources provided under the Transition Project enabled the implementation of SCRIBE in all five Storm Prediction Centres. Interviewees explained that automation ensured the consistency of forecast format across the regions and freed up the time of meteorologists needed to focus on high-impact weather events.
  • Environment Canada program staff are of the view that continued improvements to SCRIBE (e.g. to integrate other forecasting tools such as the Air Quality Health Index and to continue work on a marine SCRIBE) are not supported by a sufficient level of resources. An Environment Canada staff member expressed concern that the level of software maintenance for basic public forecast needs has been reduced.
  • While some Environment Canada interviewees point out that SCRIBE is built on a rapidly aging technology, additional secondary information indicates that the next generation of SCRIBE will build on an investment of the Transition Project in the Ninjo workstation project.³⁰

Communications Program

• Challenges were experienced in obtaining a comprehensive picture of the extent of communication relating to the Transition Project because of, for example, difficulties in locating documentation and staff turnover. Findings and examples provided are therefore based upon information found.
• Document reviews, website searches and secondary data provide examples of material used to inform MSC employees, the media and the general public about the MSC’s Transition Project. Examples include:
  • Bulletins, communiqués and annual reports were used to communicate the information about the MSC Transition Project to employees; and
  • News releases, background information and other products were used to inform the media and the general public about the project.

1 b) Intended Immediate Outcome: Improvements in warnings and forecasts of high-impact events

• Environment Canada interviewees are of the view that performance has improved over time and that these improvements were a function of the Transition Project and other A-Base programs.
• Some interviewees note the absence of performance data in this area and a need to define what is meant by a “basic level of service” and what is meant by a “high-impact” event, and to ensure a common understanding across regions.
• While other interviewees point to recent examples of the department’s success in issuing warnings of high-impact events (e.g. Post-Tropical Storm Noel), it is not clear to what extent improvements can be related to the Transition Project alone.

1 c) Intended Immediate Outcome: Improvements in technology transfer through the creation of five new national research and development labs to support Storm Prediction Centres

• Documents and Environment Canada interviewees report that five new national research and development laboratories were created and collocated with the Storm Prediction Centres.  Each lab had a different focus: Dartmouth — marine and coastal; Montreal — severe weather; Toronto — nowcasting; Edmonton — arctic weather and hydrometeorology; and Vancouver — coastal mountain meteorology.
• A challenge noted was that the announcement of the national labs was not accompanied by additional funding.
• With respect to improvements in technology transfer to the Storm Prediction Centres, interviewees provided little detail and explanation. Interviewees seemed to associate improvements in technology transfer to the length of time that the laboratory has been in operation.

Rating: Progress made, attention needed.

2 a) Intended Immediate Outcome: Improvements in services for and strengthened partnerships with clients

• Overall, Environment Canada interviewees are of the view that improvements in service and strengthened partnerships with clients will only become evident in the coming years.
  • Environment Canada interviewees note that the work of some National Service Offices has been encumbered by their geographic locations, the delays in start up due to the requirements for language training and budget cuts which impeded hiring.
• On the other hand, some Environment Canada interviewees and documentation reviewed indicate that Warning Preparedness Meteorologists improved services and partnerships by working with Emergency Management Organizations and engaging and enhancing relationships with the media.
• Interviewees from the department and other government and non-government organizations note that Environment Canada’s website has increased the availability and accessibility of data and forecasts for different government jurisdictions, media and the general public.
• EC interviewees indicate that services for clients were strengthened with the establishment of the National Inquiry Response Team, which centralized and directed feedback from the public to the appropriate departmental managers.

2 b) Intended Immediate Outcome: Increased use and effectiveness of atmospheric data and services through a new outreach network

• Information regarding the extent of achievement of this outcome was not found. Rather, secondary data and documentation focused on the formation of a new outreach network.
  • Secondary data and reports indicate that 75% of the outreach officer positions have been staffed and this has increased client awareness of products.  At the same time, Environment Canada interviewees perceive that more could have been done.
    • Environment Canada interviewees note confusion over the roles of outreach officers.

Rating: Although there is compelling subjective evidence that the program is doing well, a complete assessment cannot be done due to the lack of human resource, financial and overall performance data.

3 a) Intended Immediate Outcome: More productive workforce with the capacity to meet MSC needs

• Environment Canada interviewees note that the automation of forecast predictions increased the efficiency of MSC staff by “freeing up” the time of meteorologists, thereby enabling them to concentrate on high-impact weather.
• Documentation (e.g. program descriptions) shows evidence of training programs (e.g. the Meteorologist Operational Internship Program, the Meteorological Technologist Apprenticeship or Professional Training Program, and the Hydrometric Technician Apprenticeship or Professional Training Program) to increase the capacity of MSC staff. In the documentation reviewed
  • The Meteorologist Operational Internship Program is noted to be an entry-level training program to teach university graduates skills and knowledge needed to become operational weather forecasters and to be the first step in the Meteorologist Occupational Training Program;
  • The Meteorologist Apprenticeship or Professional Training Program is designed to address the gap between qualifications obtained by recruits prior to joining MSC and those that are required by Operational Technologists; and
  • It is stated as part of the objective of the Hydrometric Technician Apprenticeship or Professional Training Program that it will ensure availability of well qualified and trained Hydrometric Technologists, and it is noted that recruitment is based on candidates meeting specified criteria, qualifications and competencies.³¹
  These programs were intended to address demographic gaps in technical and scientific areas of expertise.
• Environment Canada interviewees note the challenges related to the restoration and development of key skill sets across the country.  For example, some staff were placed in bilingual positions that required additional language training.

3 b) Intended Immediate Outcome: Sustainable workforce

• Environment Canada interviewees and documentation indicate ongoing recruitment in the Meteorology Group (MT) and Engineering and Support Staff (EG) categories, although it is not evident the extent to which the recruitment is linked to the Transition Project as compared with other MSC programs.
• Internal and external interviewees perceive that current succession planning is not enough to address the impacts of an aging workforce.  Environment Canada interviewees are of the view that it is hard to get a sustainable workforce with financial pressures.³²

Rating: Progress made, attention needed

4 a) Intended Immediate Outcome: Improved extended range and seasonal forecasts

Three- to Five-Day Forecasts

• Interviews, secondary data and reports indicate a steady trend of improvement in the quality of the three- to five-day forecasts over the years.
  • It is noted that this improvement cannot be attributed to the Transition Project alone.

Extended Range Forecasts

• Environment Canada interviews, secondary data and reports indicate that the extended range forecast, which was to extend the prediction capability to day six and seven, is expected to be achieved in 2008. No verification of the accuracy of this product is available yet.

Seasonal Forecasts

• Interviews, secondary data and reports indicate that the Transition Project allowed for the development and implementation of an improved seasonal forecast system and that the new system became operational in December 2007. It is reported that the new system shows an improvement over the previous systems.

North American Ensemble Forecast System

• Interviews, secondary data and reports indicate that the Transition Project enabled the North American Ensemble Forecast System to be put into operation. The verification system is under development.

4 b) Intended Immediate Outcome: Improved winter road safety weather services

• Documentation reviewed indicates that the Road Weather Information System (RWIS) has been operational since February 2006. An Environment Canada staff member reports that as of the end of FY 2007–2008, Environment Canada has data-sharing agreements with British Columbia, Alberta, Manitoba, New Brunswick, Nova Scotia, Prince Edward Island, Newfoundland and Labrador, and Yukon.

4 c) Intended Immediate Outcome: Enhanced scientific analysis of high-impact vulnerabilities and adaptations

• Environment Canada interviewees and documents reviewed³³ indicate that progress has been made towards focusing on high-impact weather and towards making this information available to the public and other governments. The intent was to bring together information related to hazards from different science sources and provide it through a website format. The hazards websites in Ontario and Quebec are cited as “success stories,” however, interviewees noted that sites for other provinces were not yet at the same level.
• In discussing the achievement of this outcome for the Ontario site, it was noted in interviews that the Government of Ontario had passed a Bill whereby every municipality had to assess hazards and risk in their community. The website information that was put together was able to be used for this requirement, and Environment Canada interviewees are of the view that this information has been integrated into the province of Ontario’s emergency preparedness program.

Rating: Progress made, attention needed

5 a) Intended Immediate Outcome: Modernized monitoring networks through network rationalization

• Environment Canada interviewees are of the view that some progress has been made in modernizing networks.
  • For example, Environment Canada interviewees indicate that Transition Project funding resulted in the purchase of new equipment for 75 Reference Climate Stations (100% achieved of intended outcome).
• The rationalization (or reduction) of networks, meaning the closure of stations, proved to be a sensitive issue at the community level.  Environment Canada interviewees indicate that in the past they have used a huge volunteer network and, even though there are now more efficient uses for the funds, it is hard to close down networks, especially when they are filled with dedicated people who have been keeping records for years. The consequence of not closing networks is that they cost money to keep.
  • A review of secondary data and reports³⁴ shows the extent to which the closure of networks has been achieved:
    • Removal of equipment and remediation of surface weather stations (38% achieved of intended outcome);
    • Removal, disposal of observing facilities on ships (67% achieved of intended outcome);
    • Decommission and cleanup of hydrometric sites (86% achieved of intended outcome); and
    • Reduction of the volunteer network (network reduced from 1600 to 846 stations).³⁵
• External factors have delayed the implementation of some activities.  For example:
  • land claims negotiations have delayed the clean up and remediation of the Mould Bay and Isachsen sites; and
  • new procedural requirements for the management of property resulted in delays to the implementation of monitoring projects.

5 b) Intended Immediate Outcome: Enhanced quality assurance of and access to key atmospheric, water, ice and air quality data

• Environment Canada interviewees report that the development of the data management framework has been slower than expected.  The data management framework is a project to consolidate a number of data acquisition and processing flows for the MSC's operational network. An Environment Canada interviewee noted that progress relating to the data management framework has been delayed by substantial reductions to A-Base resources and hence the inability to staff some positions.  Another Environment Canada staff member noted that an automatic quality control intended to be a component of the data management framework has not yet been developed and is behind schedule. Environment Canada interviewees and federal partners and stakeholders are of the view that this is a serious shortcoming for the department and other stakeholders. As one interviewee explains, “having a sustainable way to manage our data is key to providing warnings to Canadians.”
• At the same time, an Environment Canada staff member notes improvements in access to data for the public.  Examples of improvements in access to information include the MSC's weather office and climate and water quantity (stream flow and level) websites.

5 c) Intended Immediate Outcome: Improved measurement of the upper atmosphere conditions via Aircraft Meteorological Data relay (AMDAR) to enable improved forecasts

• An Environment Canada staff member reports that Aircraft Meteorological Data Relay (AMDAR) systems were implemented on eight aircraft operating in remote locations. The older avionics of these aircraft, however, did not allow the AMDAR systems to generate data meeting international AMDAR data quality standards.  It was emphasized by this staff member that work done as part of the Transition Project has advanced the overall file on measuring upper atmosphere conditions, and other avenues to move forward with the file are being examined.

EQ 4 B Ultimate Outcome Findings: Success

Findings for the achievements of ultimate outcomes are mostly based on interviews with stakeholders and workshops conducted with Environment Canada employees.

Ultimate Outcome 1: A more sustainable organization and infrastructure

Rating: Little progress, priority attention required

Overall Finding

The majority of interviewees, both internal and external to Environment Canada, are of the view that MSC has not become a more sustainable organization.

Detailed Findings

• Environment Canada Employees: The majority of Environment Canada interviewees are of the view that funding provided by the Transition Project has not made the meteorological science, production, services and infrastructure more sustainable. Although some elements are in place such as a life-cycle approach to managing equipment, replacement of some equipment in the Reference Climate Stations and the automation of forecasts, interviewees are of the view that budget and human resource cutbacks are the biggest barriers to the stabilization of services.
• Federal Partners and Stakeholders:  Federal partners and stakeholders have mixed views as to whether MSC has become a more sustainable organization over the last five years.
• Provincial and Municipal Stakeholders:Provincial and municipal stakeholders had few direct comments to offer with respect to the sustainability of MSC and infrastructure.  Changes in personnel and the perception that the MSC had become more centralized were noted by interviewees.
• Non-Federal Partners and Stakeholders: Non-federal partners and stakeholders consider that Environment Canada has moved away from being a sustainable organization and infrastructure in recent years. Some non-federal interviewees perceive that MSC has become fragmented from an organizational perspective and no longer exists as a unified service.  Also, non-federal interviewees perceive that succession planning has not been adequately addressed by MSC.

Ultimate Outcome 2: Strengthened linkages between production, science and service

Rating: Progress made, attention needed

Overall Finding

Stakeholders are of the view that some progress has been made towards strengthening linkages, but more progress is needed to achieve this outcome.

Detailed Findings

• Environment Canada Employees:  Many Environment Canada interviewees consider that the building blocks (e.g. national laboratories, research support desks) are in place but that the linkages between production, science and service have not been strengthened to the extent needed.  Environment Canada interviewees express a variety of views:
  • “NSOs and outreach are good ideas, but there is further to go”;
  • “Labs are good ideas, but they do not function at the same level”;
  • “Linkages between science and production [are ]uneven”; and
  • “Linkages between science and service [have ]deteriorated".
  
  Echoing the concerns, workshop participants are of the view that the department’s Transformation Process fragmented linkages rather than strengthening them and that “the Transformation goals of centralizing authority and decision-making, and standardizing policy implementation had an impact on the whole organization, including the MSC’s organizational structure, resources, supports and program priority-setting…[m]any MSC workshop participants feel powerless, having fewer resources, less direction from senior management and less decision-making authority.”³⁶
• Federal Partners and Stakeholders:  Federal partners and stakeholders hold mixed views. While several of the interviewees were not able to comment on whether the MSC has shown movement in the last five years towards strengthened linkages, other interviewees pointed to the increase in the level of collaboration between production, science and service, as well as services being more direct and targeted as examples of improvements.
• Provincial and Municipal Stakeholders: Limited responses were received from provincial and municipal stakeholders regarding this outcome.  An interviewee commented on being impressed by MSC’s ability to connect them with people in the organization and another noted an increase in outreach.
• Non-Federal Partners and Stakeholders: Non-federal partners and stakeholders point to the establishment of research labs as being a positive outcome but are of the view that critical linkages between science/production/service have been fragmented or broken. Some of the non-federal interviewees perceive that MSC’s automation of forecasts has reduced services to the public.
• International Stakeholder:  An international stakeholder interviewed considers that progress has been made in strengthening linkages but identifies concern with the separation of research from MSC.

Ultimate Outcome 3: Improved services for Canadians and key stakeholders

Rating: Progress made, attention needed

Overall Finding

Improvements are noted by all stakeholders, but further improvements required.

Detailed Findings

• Environment Canada Employees:  Environment Canada interviewees report that improvements (e.g. high-impact weather, creation of the Canadian Atmospheric Hazards Network and information on MSC websites) have increased the availability and accessibility of information. Data indicate that the usage of MSC websites, specifically weatheroffice.ec.gc.ca and climate.weatheroffice.ec.gc.ca, has increased substantially over the Transition Project period.
  • The number of sessions recorded on the Weatheroffice website has increased from approximately 12 million in October 2004 to over 33 million in October 2007. With respect to specialized users, media groups have had free access to a dedicated website for media that allows them to tailor information to meet their individual needs, although the initiation of this website predates the Transition Project.
  • An Environment Canada interviewee identifies two primary challenges to improving services: 1) new products cannot be developed if resources are insufficient and 2) new products cannot be developed if other products are not dropped.
• Federal Partners and Stakeholders: Federal partners and stakeholders consider that progress has been made and more progress is needed to meet this outcome.
  • Examples of improvements cited include the creation of a National Service Unit agricultural office in Regina, the Road Weather Information System, an increased focus on the needs of clients and the standardization of marine weather products.
    • On the other hand, federal partners and stakeholders are concerned that Environment Canada resources may be insufficient to keep up with science and production needs.
• Provincial and Municipal Stakeholders: Provincial and municipal interviewees identify that improvements to service have been made.
  • For example, a provincial interviewee considers that in recent years MSC has given a greater priority to dealing with clients, stakeholders and the public. “Before that, EC was more insulated (likely linked to what was happening in the 90’s with reorganization/downsizing); it was not a very good period in terms of partnering. This is driven by the fact that we are looking at climate change scenarios in the country as a whole and need governments to be working together around those issues.”
• Non-Federal Partners and Stakeholders: Non-federal partners and stakeholders note that the increased availability of data has improved services to stakeholders and impacted their research.  However, interviewees also note concerns regarding service provision (due to staffing issues) and high prices charged to clients.

Ultimate Outcome 4:  Adaptation of Canadians in ways that safeguard security, health and safety, economic prosperity, and environmental quality

Rating: Not applicable

Overall Finding

Information available at the time of the evaluation indicates that, for the most part, it is too early to tell whether the Transition Project has helped Canadians to protect their well-being,  economic prosperity and environmental quality. Moreover, a change in the behaviour of Canadians would be difficult to attribute to the Transition Project and Environment Canada alone (e.g. the Road Weather Information System is not solely an MSC Transition Project and Environment Canada responsibility).

EQ 6 Findings

• Few unintended (positive or negative) outcomes were identified. A notable example of a positive unintended outcome arising from the project is the creation of the National Inquiry Response Team, which used staff from the National Service Offices.  The team tracks telephone and email inquiries from stakeholders and the public and directs them to the responsible manager for action, thus coordinating the response process.  An interviewee noted that the creation of this team was not part of an intended outcome of the Transition Project, rather it was a spin-off.

4.3 Cost-Effectiveness/Alternatives

Overall Findings

• Although there is compelling subjective evidence that the project provided value for the money received, a complete assessment could not be done due to the lack of financial information.
• While there are no obvious benchmarks to learn from, Environment Canada employees widely agree that the project used the most appropriate and effective means of achieving intended outcomes.

EQ 7 Findings

• There is a widespread perception among departmental staff and federal partners that the MSC Transition Project provided good value for money given the circumstances surrounding the project. Overall, Environment Canada employees agreed that the activities were the most appropriate way to achieve the desired outcomes.
• Environment Canada interviewees noted several external factors that affected the achievement of outcomes:
  • The ability to achieve outcomes was affected by department-wide changes to governance and resource allocation processes.  Budget decisions became the responsibility of Boards, not line-based organizations, and resources were centralized and allocated on a priority basis. Environment Canada interviewees note that changes in resource-allocation processes and departmental cutbacks meant that the monitoring component did not always have access to its A-Base and capital resources.
  • Government decisions were not matched with funds (e.g. the cost of the offices in Gander, Newfoundland and Labrador, and Winnipeg, Manitoba) and resources had to be re-directed from Environment Canada’s A-Base programs.
  • The implementation of activities was delayed by policy requirements (e.g. land claims negotiations) and new procedures for the management of property.
• Environment Canada employees and provincial and international stakeholders cite examples as to how the Transition Project leveraged other resources. Internally, Environment Canada employees indicate that many of the initiatives of the Project accelerated work on existing MSC initiatives.  Externally, the project leveraged resources provided by Transport Canada in the Road Weather Information System and in the implementation of the North American Ensemble Forecast System.
• Financial breakdowns for each component are presented in Annex G. While there is widespread agreement that the department received good value for the money, this perception could not be substantiated with financial information provided to date. Even though Environment Canada interviewees put extensive efforts into accounting for the use of funds, 17% of the Treasury Board Secretariat funding allocation has not been accounted for. Environment Canada interviewees attribute this difficulty to: changes made to departmental resource allocation processes post 2004³⁸; the inconsistency in coding expenditures³⁹ and transfer of Transition Project funds to other Boards.⁴⁰  As of the writing of this report, the use of around $5.36M of Treasury Board funds cannot be tracked to expenditures made under the Transition Project.⁴¹ Further, $7.3M allocated for the decommissioning of Mould Bay and Isachsen could not be accounted for.⁴² The total of these two figures amount to 17% of the funds allocated by Treasury Board for the Transition Project.

EQ 8 Findings

• At a general level, Environment Canada interviewees note that the strategy was appropriate, although some did feel that there should have been more structure than what was undertaken.
• While examples from other countries show alternative ways of structuring and governing the delivery of meteorological services, no clear evidence was found that such alternative means would be more effective in achieving the Transition Project’s intended objectives, given Canada’s unique geography, population size, type of governance, and budget.
• Although the governance and organizational structure of other countries’ meteorological services were examined, no formal benchmarking study was initially produced to inform and justify the Transition Project’s design and no documentation was found to explain the rationale for the Transition Project design choices.

4.4 Design and Delivery

Overall Findings

• For the most part, the concept and design of the project were appropriate to achieving the intended outcomes, although it was not delivered entirely as planned.
  • Some activities were dropped and some have not been completed to date, notably in the monitoring component.
  • Pockets of dissatisfaction with the project exist among Environment Canada interviewees and some stakeholders. Some Environment Canada interviewees are concerned about the potential to achieve the outcomes sought, and some stakeholders express concern over the delivery and quality of services.
  • MSC underestimated some of the funding requirements and resources were re-allocated within the project.
  • As a result of the inability to access all financial information, the evaluation was not able to confirm whether the project remained within budget.

EQ 9 Findings

• Environment Canada interviewees and workshop participants⁴³ generally agree that the concept and design of the project were appropriate to achieve the desired outcomes. Some interviewees noted, however, that certain sub-components were problematic.
  • Environment Canada interviewees, secondary data and reports indicate that some activities were only partially completed (e.g. decommissioning of sites and the development of quality assurance programs).
  • Environment Canada interviewees and documents indicate that some monitoring activities were dropped (e.g., the water vapour data acquisition program) or delayed (e.g. Mould Bay and Isachsen sites).
  • Resources had to be re-allocated, and some funds were not used.
  • Resource allocations for each component for the five-year period of the Transition are presented in Annex A. Detailed financial analyses (i.e. budgeted vs. actual expenditures) for each component are presented below.

• Expenditures amounted to approximately $20.1M for the first four years of the Transition Project, whereas the original five-year budget for this component amounted to $13.068M (53.8% actual over budget).
• Consolidation and modernization of the MSC’s forecast operations ran significantly over budget (the costs to complete the automation and consolidation activities of this component of the Transition Project were under-estimated).
• Specifically, the sub-component “Consolidation of Weather Forecast Production Capabilities” has run approximately 61% over budget, and the sub-component “Automated Routine Forecast Production” has run approximately 72% over budget (figures are based on original five-year budget allocated to these two sub-components).
• Resources were re-allocated from Component 3: Restoring and Developing Key Skill Sets.
• Only about 16% of Transition Project funds allocated for communications have been spent thus far.

• Of the $12M allocated by Treasury Board Secretariat to this component, 70.7% ($8.5M) have been spent thus far.
• Although all three National Service Offices are essentially staffed and operational, there are some current vacancies due to recent retirements, people moving on to other positions, etc. which could account for part of the difference between budgeted and actual (during the period between FY 2003–2004 and 2006–2007, $9.6M was budgeted for this component; only approximately $8.5M was actually spent).

• $3.192M was spent in the first three FYs of the project out of a total budget of $11.66M.
• Essentially all of the remaining resources were re-directed in FY 2004–2005 to automation and consolidation (as part of the Consolidation and Modernization of the MSC’s Forecast Operations component) as these activities ran well over budget.
• However, starting in 2004–2005 through the MSC Focusing for the Future initiative, funding for recruitment was built into A-Base funding and added to resources for managers.

• $11.6M had been allocated for this component by Treasury Board Secretariat. Thus far, in the four years between FY 2003–2004 and 2006–2007, approximately $8.3M (71.5%) has been spent.
• Only 67.1% ($737,600) had been spent thus far compared to the overall amount of funding ($1,100,000) allocated to the sub-component “High Impact Weather and Climate.”

Actual expenditures for “Extended Range and Seasonal Forecasts” were on target with budgeted amounts, while 73.3% of funds allocated for the Road Weather Information System between FY 2003–2004 and 2006–2007 were spent.

• Overall, $15M (58.9%) has been spent of the $25M allocated to this component by Treasury Board Secretariat.
• With regard to the sub-component, “Rationalizing the Monitoring System,” $17.11M had been allocated over the five-year period of the Transition Project. Of this amount $8M (of which $700,000 has been spent thus far) had been allocated to two specific projects — the decommissioning of Mould Bay and Isachsen.  As a result of aboriginal land claims that came to light with regards to these two sites in 2004–2005 and 2005–2006, a decision was made to delay these two projects (they have subsequently become the responsibility of Assets Management under Contaminated Sites Management). Therefore, of the remaining $9.11M, $8.925M has been spent (approximately 98%).
  • The $7.3M allocated for Mould Bay/Isachsen cannot be accounted for.
• About 81% has been spent to date on Assuring Data Quality and Access, which includes the development of a life-cycle management support system and climate and water data archive capacity.
• Only 32% has been spent thus far in comparison to budgeted amounts up to FY 2006–2007 on Aircraft Meteorological Data Relay (AMDAR) system.

EQ 10 Findings

• Detailed performance information at the activity level exists and is used particularly at a managerial level to improve products and services. Internal interviews and the document review indicate a wide variety of information.
  • Examples include: the number of hits on MSC’s websites are tracked; the National Inquiry Response Team collects public feedback and provides it to the Outcome Project lead for action; product surveys, human resources surveys and public opinion surveys have been conducted; verification systems are in place or are being developed to measure accuracy and timeliness of forecasting models and forecasts; and more recently, aspects of the Transition Project have received ISO 9000 certification.
• Although performance measures and a performance measurement strategy were approved by the Assistant Deputy Minister of MSC in 2005, performance information is rarely linked to outcomes at the immediate and ultimate outcome levels. Further, it is difficult to track/trace resource budgets versus expenditures particularly after 2005. This means that the department is unable to track up to 17% of the funds allocated by Treasury Board Secretariat.
• In the past, ad hoc reports were prepared in response to requests of management committees or boards. Now with ISO certification, management reviews are being conducted and the results are presented to the WES Board on a regular basis.

Overall Findings

Environment Canada staff voice satisfaction with specific Transition Project aspects, as well as concern with the ability or capacity to achieve outcomes of the project. The external stakeholders were generally satisfied with the relationships but express concern about the ability of MSC to provide data that stakeholders rely on, about the lack of media access to a forecaster at the local level, and about the turnover of personnel and perceived lack of succession planning. The general public appears to be satisfied with MSC's services except in local situations where regional offices were closed down.

Detailed Findings

Environment Canada Employees

• Environment Canada employees initially supported the vision, although there was some dissatisfaction with management’s handling of the Transition Project.  Although some satisfaction with specific Transition Project aspects was expressed, Environment Canada program staff voice concern over the ability to achieve outcomes and the need for management direction.⁴⁵
  • The results of the 2003 and 2004 Transition Employees Surveys indicate that MSC employees initially supported the vision, although there was some dissatisfaction with management’s handling of the Transition Project.
  • While the results of the 2007 Transition Employee Survey have not been released to date, interviews conducted under this evaluation show that MSC and internal partners⁴⁶ are generally satisfied with management’s handling of the Transition Project but are of the view that management could have done a better job of handling priorities to make sure that goals were met.
  • Environment Canada program staff interviewees expressed the view that Human Resources’ handling of the Transition Project was very good.
  • Some Environment Canada program staff interviewed were satisfied with the training provided while some staff expressed concerns.  As one Environment Canada interviewee noted, “while [MSC] tried there, I think we’ve failed in the sense that I don’t think we had the capacity to allow staff, particularly forecast staff, to take advantage of what we promised…to allow 20% of [the staff’s] time to develop and improve scientific skills."
• Mixed responses were noted by Environment Canada program staff interviewees regarding the handling of relationships between partners and stakeholders.  Respondents noted that for the most part partners seem happy; however, frustration that data was not being provided to partners who rely on the data was expressed.  When commenting on satisfaction with the handling of the relationship between the department and the general public, Environment Canada program staff interviewees felt that the general public was largely unaffected by or unaware of the Transition Project, except where a regional weather forecast centre was to be closed in their community.
  • For example, public concern over the closure of the weather forecast office in Gander contributed to the re-opening of weather forecast operations.

Federal Partners and Stakeholders, Provincial and Municipal Stakeholders

• Federal partner and stakeholder interviewees generally express satisfaction with their relationships with MSC, particularly in terms of communication, cooperation and consultation.  Federal partners and stakeholders are concerned, however, that data is not as available or timely as needed.
• Provincial and municipal interviewees note that new initiatives and partnerships are on the rise and are seen as improving in recent years.  Increased data sharing and collaboration was also noted in the interviews.

Non-Federal Partners and Stakeholders

• Non-federal partner and stakeholder interviewees express satisfaction with relationships created over the past five years and dissatisfaction with certain gaps in service. For example, some non-federal interviewees note that the information available on the Internet serves the general public but not specific industry needs.  Comments were also made regarding a lack of media access to a live local person within the MSC for severe weather information and the charging for the provision of MSC weather information on media websites.
  • Non-federal partner and stakeholder interviewees also express concern regarding turnover in personnel and are of the view that succession planning is not being addressed.
• Regarding the handling of the relationship between Environment Canada and the general public, non-federal partner and stakeholder interviewees expressed the point of view that relationships are generally good; however, there were some negative comments, such as the sense of connectivity at the community level being lost due to the closing of the regional weather offices.

International Partners and Stakeholders

• In an interview with an international partner and stakeholder, satisfaction with the relationship was expressed and the relationship was described as open and accommodating.

• Of the 29 articles on the MSC Transition Project identified by the media scan, 17 articles can be seen as negative press on the MSC Transition Project, while 12 articles report positively on Transition Project activities. The most negative aspect of the Transition Project in the media scan is in regard to the weather station closures and the impact on MSC employees and residents living in those areas where a weather station was closed.
• According to a survey, 75% of respondents are satisfied or extremely satisfied with the media website, with 59% of respondents visiting the site more than once per day.⁴⁷

The findings related to best practices and lessons learned are based on an analysis of findings for the previous 11 evaluation questions.  The results of this analysis have been grouped into best practices and lessons learned.  Best practices are aspects that have contributed to the achievement of results, while lessons learned identify aspects that have detracted from the ability to demonstrate the achievement of results.  Each of the best practices and lessons learned described in this section speaks to the design and delivery of the MSC Transition Project.  These lessons learned could also be applicable to the design, implementation and monitoring of other projects introduced to accelerate the achievement of A-Base programs and goals.

Best Practices

• A clear vision facilitates employee support for a new initiative. Employees bought into the vision for the Transition Project and continued to be supportive of the vision throughout the duration the project.
• A logic model and performance measures provide a base upon which a project can be assessed. In 2005, MSC developed and approved a framework (logic model and performance measurement strategy). This framework enabled the reporting of ultimate outcomes and provided a structure by which the project could be evaluated.

Lessons Learned

• Findings from the evaluation show that the roles and responsibilities of the National Service Offices and outreach officers were not clearly understood. It is important to ensure aspects and roles specific to an initiative are clearly defined and understood even if they evolve over time.
• Findings also show that Environment Canada employees and external stakeholders, particularly non-federal stakeholders, did not understand the impact of changes created by the 2005 departmental transformation process on the mandate and services of MSC. As a result, these stakeholders held the view that MSC was becoming a more fragmented and less stable organization.  Communication strategies play a key role in shaping stakeholders’ perceptions of an initiative.
• Project managers underestimated the costs of some activities,⁴⁸ did not anticipate the role and impact of internal and external factors, and were overly optimistic about the ability to achieve immediate and ultimate outcomes.
• While the Transition Project was able to track the expenditures of the Transition Project by creating a separate Outcome Project under the Results Management Structure, this mechanism was not sufficient to determine the use of Treasury Board Secretariat and internally reallocated funds. Methods of tracking project specific financial data and linking it to a larger entity (e.g. ongoing A-Base programs) are essential to determine the value for money received.
• The logic model for the Transition Project identified ambitious outcomes for a five-year project. Many of the outcomes of the project, particularly those related to the introduction of new products and services, take time to manifest.  Accordingly, logic models for short-term projects need to be realistic in terms of timeframes.
• In 2005, a performance measurement and an evaluation strategy were introduced through the development of an evaluation framework. Yet many of the Environment Canada interviewees were not aware of the performance measures and information requirements for an eventual evaluation. Thus, a great deal of time was taken to collect performance information for the evaluation. The performance measurement strategy that was developed needs to be implemented. A performance measurement strategy is only useful if efforts are taken to monitor the progress of the initiative against the performance measurement strategy. Methods of tracking performance data, especially when the project is part of a larger entity, are essential to determine the extent of achievements of a project.

²⁵ As discussed in section 3.2, ratings have been assigned to many of the evaluation questions.  Possible ratings include Achieved; ~Achieved; Progress made, attention needed; and Little progress made, attention needed.  When it has been determined that it is too early to assign a rating, Too early to say is indicated. Not applicable has been indicated for items when a rating has been determined not to apply.  See section 3.2 for further detail. 

²⁶ Budgets reviewed included Budget 2003, Budget 2004, Budget 2005, Budget 2006 and Budget 2007.  Examples of topics include infrastructure, emergency preparedness and response, and skills and learning.

²⁷ Speeches reviewed included Speeches from the Throne 37^th Parliament, 2^nd session to 39^th Parliament, 2^nd session.  Examples of topics include infrastructure, emergency management, training and skill development.

²⁸ Goss Gilroy Inc., Report on Workshops for the MSC Transition Project Evaluation, December 6, 2007, p. 14.

²⁹ Data taken from “How MSC’s Financial Situation has Evolved,” September 30, 2004 and other internal documents prepared for the WES Board.

³⁰ Ninjo is a workstation software.  A workstation is the computer used by the forecaster to access, display and manipulate data.

³¹ Details provided regarding the programs are based on information contained within the program description documents obtained from the MSC.

³² For more information on financial pressures, see the findings for EQ 7.

³³ E.g. MSC Year 3 Transition – Year End Report; www.hazards.ca; www.criacc.qc.ca

³⁴ Based on information received on January 15, 2008.

³⁵ The number of volunteer networks was increased from the initial intended outcomes. Thus, the achievement is not calculated as a percentage.

³⁶ Goss Gilroy Inc., Report on Workshops for the MSC Transition Project Evaluation: Final Report, p. 5.

³⁷ ~Achieved:  Although there is compelling subjective evidence that the program is doing well in the given issue area, a complete assessment cannot be done due to lack of performance and financial data.

³⁸ Allocation of resources was based on priority requirements.

³⁹ The fact that managers can code the use of funds differently means that errors in coding may have occurred. As such, researchers were only able to derive a rough approximation of the total value.

⁴⁰ For example, funds for core support services were transferred to the Departmental Management Services Board, but the use of these funds was not necessarily linked back to the Transition Project itself.

⁴¹ $73.8M was received from Treasury Board. Of this amount: $56,260,300 was spent between FY 2003–2004 and FY 2006–2007 on program activities (includes $1,189,000 allocated for Accommodations expense for five years); ~$5.36M cannot be accounted for.

⁴² The decision was made to delay these two projects (they have subsequently become the responsibility of Assets Management under Contaminated Sites Management).

⁴³ Goss Gilroy Inc., Report on Workshops for the MSC Transition Project Evaluation: Final Report, December 6, 2007.

⁴⁴ External stakeholders did not know specifically about the Transition Project.  In order to get their input, external interviewees were asked questions about 1) elements of the Transition Project that related to their area of work and 2) their relationship with Environment Canada over the duration of the Transition Project period (2003–2004 to 2007–2008).

⁴⁵ Goss Gilroy Inc., Report on Workshops for the MSC Transition Project Evaluation: Final Report, December 6, 2007.

⁴⁶ Internal partners refer to Environment Canada staff within the Science and Technology and Environmental Stewardship branches.

⁴⁷ MSC Media Website Survey Final Report, July 2007.

⁴⁸ For example, the cost of consolidating and modernizing forecast operations.