Home > Proactive Disclosure > Audits and Evaluations > A&E Reports > 2008-2009 > Evaluation of the Meteorological Service of Canada (MSC) Transition Project

Introduction:

The Audit and Evaluation Branch of Environment Canada is conducting an evaluation of the Meteorological Service of Canada (MSC) Transition.  The purpose of this evaluation is to assess the relevance, success, cost-effectiveness, and design and delivery of the MSC Transition project for the five year period from 2003/4 to 2007/8.

As one component of the research, we are conducting key informant interviews in order to obtain a range of perspectives from those who are or who have been involved in this project.  The questions below serve to guide this interview process.

Overview Question:

1. Briefly describe the nature of your involvement/role within the MSC Transition project. How long have you been involved?
   1. What particular component(s) of the Transition have you been most involved with?
      The components of the MSC Transition project are:
      • Consolidating and modernizing MSC’s forecast operations
      • Creating National Service Offices and outreach capacity
      • Restoring and developing key skill sets
      • Introducing product and service enhancements and innovation
      • Invigorating the MSC’s monitoring capacity
   2. Within this particular component, who were the primary stakeholders and beneficiaries with whom you were involved?
      
      Prompt: MSC Staff, Canadian Public, OGDs, Private Sector, NGOs, Media, Provincial and Municipal Governments, and International Partners


2. Briefly describe your interaction with the Meteorological Service of Canada over the last 5 years.
   1. Are you aware of the MSC Transition project? If yes, please describe your understanding of this project.
      
      Prompt: The intention of the MSC Transition funding was to allow the MSC to transition to a modern, sustainable organization providing quality services to Canadians.  See Annex 1 for a list of Transition components.

Relevance Questions:

3. To what degree was/is the MSC Transition project aligned to:
   1. Departmental strategic outcomes?
   2. WES Board/WEP Table/MSC priorities?
   3. Government of Canada priorities?
      Please describe. (REF EQ1)


4. The intention of the MSC Transition funding was to allow the MSC to transition to a modern, sustainable organization providing quality services to Canadians. In order to do this 5 needs/areas were identified, these include:
   • Consolidating and modernizing MSC’s forecast operations
   • Creating National Service Offices and outreach capacity
   • Restoring and developing key skill sets
   • Introducing product and service enhancements and innovation
   • Invigorating the MSC’s monitoring capacity
     
     In your opinion, to what degree has the MSC Transition project addressed this need?
     Please describe. (REF: EQ2)


5. How do the Government of Canada’s meteorological research and development, products and services currently contribute to or advance:
   1. The work of your organization?
   2. The international meteorological community?
      Please explain. (REF EQ2)


6. In 2004, Environment Canada began a Transformation from a line-based organization to a results-based management structure. This Transformation integrated common departmental functions under a new Board structure. This resulted in the shift from WEP Table to WES Board.
   
   What impacts, if any, have changes in the governance structure of the Department had on the MSC Transition project? Please describe. (REF EQ3)

Success Questions:

7. You mentioned during question 1 that you had been most involved with the following component(s) of the MSC Transition project:
   To what degree have the activities and outputs associated with this/these component(s) contributed to the achievement of their intended outcomes? (See Annex 1) (REF EQ4)



8. To what degree have the outcomes associated with this/these component(s) (outlined above) contributed to the achievement of the following final outcomes of the MSC Transition project?
   1. A more sustainable organization and infrastructure;
   2. Strengthened linkages between production, science, and service;
   3. Improved services for Canadians and key stakeholders;
   4. Adaptation of Canadians in ways that safeguard security, health and safety, economic prosperity, and environmental quality? (REF EQ4)


9. Overall, to what degree has the MSC Transition project achieved the following final outcomes:
   1. A more sustainable organization and infrastructure;
   2. Strengthened linkages between production, science, and service;
   3. Improved services for Canadians and key stakeholders;
   4. Adaptation of Canadians in ways that safeguard security, health and safety, economic prosperity, and environmental quality? (REF EQ4)


10. In the last 5 years, to what degree has the MSC demonstrated movement toward achieving the following Transition outcomes:
    1. A more sustainable organization and infrastructure;
    2. Strengthened linkages between production, science, and service;
    3. Improved services for Canadians and key stakeholders;
    4. Adaptation of Canadians in ways that safeguard security, health and safety, economic prosperity, and environmental quality? (REF EQ4)


11. What external factors (outside of the MSC Transition project), either positive or negative, have influenced the MSC Transition project, its implementation, and/or the achievement of its outcomes?
    
    1. What have been the impacts of these external factors?
       
    2. Were these factors addressed? Please describe. (REF  EQ5)


12. Have there been any unintended/unexpected outcomes, either positive or negative, resulting from the MSC Transition project?  Please describe. (REF EQ6)

Cost-Effectiveness/Alternatives Questions:

13. To what degree do you feel the MSC Transition project provided value for federal dollars spent?  Was Transition funding allocated and used appropriately in order to address MSC/WES needs?  Please describe with examples. (REF EQ7)



14. Was adequate funding (Treasury Board and internal reallocation) provided to each of the five components of the MSC Transition project?
    Components include:
    • Consolidating and modernizing MSC’s forecast operations
    • Creating National Service Offices and outreach capacity
    • Restoring and developing key skill sets
    • Introducing product and service enhancements and innovation
    • Invigorating the MSC’s monitoring capacity (REF EQ7)


15. Given the component(s) you are most involved with, to what degree did the MSC Transition project use the most effective approaches/strategies to achieve its outcomes? Please describe.
    1. Are there alternative approaches/strategies that would have made the MSC Transition project more effective? Please describe. (REF EQ8)

Design and Delivery Questions:

16. To what degree has the MSC Transition project been delivered as designed/intended?  Please describe aspects of the Transition project that were not delivered as designed. (REF EQ9)



17. Was the MSC Transition project completed within its identified budget?
    
    1. How were MSC Transition project priorities determined?
    
    
    2. Did reallocations occur as originally planned? 
    
    
    3. Were additional reallocations required?  If so, please describe. (REF EQ9)


18. Has performance data been collected and reported against MSC Transition project activities/outcomes? Please provide specific examples.
    
    1. If yes, how has the information been used? (REF EQ10)


19. As an MSC employee, to what extent are you satisfied with the following aspects of the MSC Transition project?
    
    1. Management’s handling of the Transition
    
    
    2. Human Resource professionals’ handling of the Transition
    
    
    3. Information regarding and availability of training and development required in order to fill new roles (REF EQ11)
       Please Describe.


20. During the MSC Transition project, how do you feel relationships with the following groups were handled?
    1. Management and staff
    
    
    2. Management and partners/stakeholders
    
    
    3. Environment Canada and the general public (REF EQ11)
       Please identify any issues that arose and if/how they were handled.


21. Over the last 5 years, to what extent has the MSC been successful in handling its relationships with: 
    1. Partners and stakeholders
    
    
    2. The general public (REF EQ11)


22. What are some examples of best practices and/or lessons learned from the MSC Transition project? (REF EQ12)

Concluding Questions:

23. In your opinion, what are the key accomplishments and challenges of the MSC Transition project? (REF EQ12)



24. Over the last 5 years, what are the key accomplishments and challenges of the MSC as they relate to your organization?(REF EQ12)



25. Do you have anything additional to add?

1. Consolidating and Modernizing MSC’s Forecast Operations
   1. Improved efficiency through consolidation of forecast operations
   2. Improved warnings and forecasts of high impact events
   3. Improved technology transfer through the creation of five new National research and development labs to support Storm Prediction Centres


2. Creating National Services Offices and Outreach Capacity
   1. Improved services for and strengthened partnerships with clients
   2. Increased use and effectiveness of atmospheric data and services through a new outreach network


3. Restoring and Developing Key Skill Sets
   1. More productive workforce with the capacity to meet MSC needs
   2. Sustainable workforce


4. Introducing Product and Service Enhancements and Innovation
   1. Improved extended range and seasonal forecasts
   2. Improved winter road safety weather services
   3. Enhanced scientific analysis of high-impact vulnerabilities and adaptations


5. Invigorating the MSC’s Monitoring Capacity
   1. Modernized monitoring networks through network rationalization
   2. Enhanced quality assurance of and access to key atmospheric, water, ice and air quality data
   3. Improved measurement of the upper atmosphere conditions via AMDAR to enable improved forecasts

Note: Budgeted and actual expenditures represent Treasury Board Secretariat funds; these figures do not include $11.25M in internal re-allocations agreed to by the department as part of the submission agreement. These figures also do not include amounts for FY 2007–2008, as these were not yet available.

⁴⁹ Evaluation Framework for the Meteorological Service of Canada Transition, April 2005.

⁵⁰ In a discussion at the WES Board meeting held February 5, 2008, members emphasized that the initiative is a project, not an ongoing program. Issue and evaluation question wording has therefore been modified from wording used in the evaluation plan to reflect this point.

⁵¹ Refer to Annex B for the evaluation issues and questions.

⁵² The master list was used to prepare interview guides for each of the interview groups.