Evaluation of the Hydrological Service and Water Survey

This report presents the results of the evaluation of the relevance and performance of the Hydrological Service and Water Survey (HSWS) program (sub-program 1.2.3 of the departmental Program Alignment Architecture (PAA) conducted by Kelly Sears Consulting Group and Environment Canada’s Evaluation Directorate, Audit and Evaluation Branch, between January and November 2013. The evaluation covered the five years from 2008-2009 to 2012-2013.

This evaluation is part of Environment Canada (EC)’s 2012 Risk-based Audit and Evaluation Plan which was approved by the Deputy Minister. The evaluation was conducted in order to meet the coverage requirements of the Treasury Board of Canada Policy on Evaluation, which require that all direct program spending be evaluated at least once every five years.

1. Program Profile

The Hydrological Service and Water Survey (HSWS) consists primarily of the Water Survey of Canada (WSC), which is the federal component of the National Hydrometric Program (NHP). The Water Survey of Canada (WSC) is overseen by EC’s Meteorological Service of Canada (MSC). In addition, sub-program 1.2.3 includes the Hydrological Service, a smaller area of activity of the MSC which involves hydrological science, applications of hydrological data (including modeling) and related services.

The NHP is responsible for collecting, interpreting and disseminating standardized information about surface water levels and flows (water quantity) across Canada. Under the authority of the Canada Water Act, the federal, provincial and territorial governments entered into bilateral agreements, established in 1975 and currently being updated, to manage the funding and provision of water quantity monitoring services, on a cost-shared basis. The WSC operates 2,783 water gauging stations in partnership with the provinces, territories and other agencies, and maintains a database containing historic data from an additional 5,577 inactive stations for the country. Each water gauging station is designated as either federal, federal-provincial/territorial or provincial/territorial, according to national classification guidelines agreed to by all parties. The federal government pays for the operational costs and recovers the appropriate share from each party based on the station designations.

The WSC’s hydrometric data are used in the analysis, modelling and forecasting of water flows and levels, and such information is used as an input to the design and management of water-related activities in a wide range of sectors. Examples of areas of application include flood risk management, emergency response management, water resources planning, water allocation, infrastructure planning and design, environmental monitoring and management, analysis of climate change and long-term weather effects, power generation and recreational uses of waterways.

Total annual expenditures by the HSWS during the period of the evaluation peaked at $35.4 million in 2009-10 and fell in each of the subsequent years, to $32.4 million in 2012-13. Over 40% of these costs were recovered from Federal/Provincial/Territorial (F/P/T) partners each year, and the amounts recovered increased steadily, from $13.3 million in 2008-09 to $15.5 in 2012-13.

2. Evaluation Methodology

The methodology for the evaluation involved:

• A review of program documents and performance data.
• A literature review of the rationale for, and benefits of, public sector delivery of hydrometric programs, and the design of comparable programs in selected other jurisdictions.
• Key informant interviews with 58 representatives of the program at headquarters and in the regions, F/P/T partners and secondary users of hydrometric data in the public, private and university sectors.

3. Findings and Conclusions

3.1 Relevance

All lines of enquiry found there to be a continuing need for timely, consistent and reliable hydrometric data, and that the demand for these data is increasing. Factors that contribute to the growth in demand include:

• An increased frequency and severity of extreme weather events, with implications for flood planning and emergency management.
• The complexity of climate change analysis.
• Increased rates of resource development, particularly in remote areas.
• Population growth giving rise to needs for new and renewed infrastructure.
• More intense use of agricultural land and water resources.

Responsibility for water resources is shared between federal, provincial and territorial governments. Federal collection and provision of hydrometric data supports the federal responsibilities for navigable waters, fisheries, international and intra-national boundary waters, federal lands, federal facilities, and First Nations reserves as well as duties under the Fisheries Actand Canadian Environmental Protection Act. Provincial and territorial governments have lead responsibility for water management and protection within their respective boundaries, and have requirements for hydrometric data similar to those of the federal government.

The Canada Water Act provides an enabling framework for the NHP, under which the WSC collects, interprets and disseminates hydrometric data on behalf of the federal, provincial and territorial signatories to bilateral agreements. Federal or shared F/P or F/T gauging stations in the national network (approximately 46% of the total) respond directly to federal government needs and priorities for hydrometric data. There is, however, some flexibility in the application of the Canada Water Act and the 1987 Federal Water Policy as to what constitutes federal obligations with respect to hydrology monitoring in Canada. Provision of hydrometric data by the HSWS aligns with EC’s Strategic Outcomes relating to conservation and restoration of Canada’s natural environment and enabling Canadians to make informed decisions on changing weather, water and climate conditions. These departmental outcomes are intended to contribute to the Government of Canada outcome, A Clean and Healthy Environment.

3.2 Performance – Efficiency and Economy

The design of the Hydrological Service and Water Survey program (hereafter referred to as “the program”) is widely considered to be appropriate for achieving its intended outcomes. The program’s design and delivery focuses on a set of core activities that ensure appropriate hydrometric data are made available for use in water management decisions. These core activities and supporting systems, such as the Quality Management System (QMS), are consistent with published commentary regarding the requirements for an effective hydrological monitoring program.

Separation of responsibilities and budgetary authorities between headquarters and the regional operations of the MSC presents a challenge for the integration of HSWS planning and management. More broadly, at the level of the NHP, the WSC functions as a national organization working on behalf of the F/P/T partners to satisfy the separate and joint needs of the partners in a cost-effective way. This approach enables national and P/T priorities and directions to be integrated. The NHP governance structure is widely perceived as being particularly effective, in that it provides a national framework to integrate national and P/T priorities while allowing for flexibility in arrangements with individual provinces and territories. Data collection and provision by a single national service provider enables economies of scale that could not be achieved by independent F/P/T entities. Also, and perhaps more importantly, because water flow is not directly measured and requires hydraulic interpretation, a single agency approach allows for consistency in estimation techniques, systematic improvements to the engineering and national standards for comparison across provinces and territories.

The introduction of digital data collection and telemetry technologies, and the deployment of the new hydrometric workstation for managing the production and interpretation of data are transforming the way hydrometric data are collected, interpreted and disseminated. The majority of the 1,875 real-time stations (67% of the stations in the NHP network) are capable of producing preliminary real-time data within 2-3 hours of initial data capture at these stations. These changes are having a major impact on the way the WSC operates and are expected to lead to improvements in the efficiency of the program as it completes the current transition.

Four areas of weakness were identified in the efficiency and economy of the HSWS:

• Gaps in the program’s professional (engineering and scientific) staff and management/ supervisory ranks are limiting the program’s ability to produce the required hydrological reports, meet transboundary commitments, undertake critical data analysis projects, and provide support for operational activities.
• Program managers and staff have limited direct contact with secondary users, outside of major users such as the F/P/T partners, and do not have a good understanding of current and emerging needs amongst secondary users of hydrometric data, how and why they are using these data, and overall trends in demand. The role of liaison with the stakeholder community is currently being reformulated within the MSC and is largely the responsibility of performance and planning groups in the MSC, not the HSWS.
• Key performance measures and indicators for the program’s principal outputs and outcomes have been defined but the program is not producing regular performance data to report on program-wide performance and does not have a logic model that could aid the selection or confirmation of performance measures and indicators.
• The program has not been able to fully achieve its target of delivering validated, archived water level and flow data within four months of the end of each calendar year.

3.3 Performance – Effectiveness

For the purposes of the evaluation, performance against three intended outcomes was assessed. In the absence of a program logic model, these outcomes were taken from the Expected Result in the Performance Measurement Framework for the PAA and the objectives of the QMS.

1. Canadians and their institutions have the hydrological data, information and knowledge they need to make water management decisions. There are two dimensions to this outcome – performance of the current network and the extent to which the network density (number of stations and their locations) is sufficient to satisfy user needs. Users of data from the current network of gauging stations are highly satisfied with the quality and completeness of the available data, particularly the real-time data. Users of archived data were satisfied but would like to see more timely production of such data.
   
   At the same time, however, studies of network adequacy and feedback from the key informants indicate that network density is not keeping up with demand. The most recent such study, commissioned in response to a 2010 Commissioner of the Environment and Sustainable Development (CESD) audit recommendation to determine the optimum number of stations, compared the current network density of the NHP to World Meteorological Organization (WMO) guidelines. This study, published in June 2013, concluded that about 12% of Canada’s terrestrial area is adequately covered by the existing network of stations, 49% is poorly gauged and about 39% is ungauged. The study’s authors estimated that over 5,000 additional stations could potentially be required.^{Footnote 1}Further work is necessary to develop a more refined assessment of optimal network density and gaps, taking into account additional parameters that were not included in the WMO physiographic units, such as ecological, social and economic factors that influence network planning. Analysis of this kind, in combination with a detailed analysis of the current and emerging demand for hydrometric data, could enable the program partners to establish network priorities.
   
   
2. A credible, sustainable national hydrometric monitoring program is maintained and supported. The program was generally perceived to be credible by many of the program and partner representatives, in that it is producing data that satisfy quality assurance requirements and are consistent and comparable over time and between different parts of the country. The results of International Organization for Standardization(ISO) audits also indicate that the program’s QMS is meeting ISO certification requirements. The ability of the program and its F/P/T partners to maintain this credibility will depend, in part, on the extent to which the network of stations can be optimized and developed in response to current and expected future demand for hydrometric data.
   
   
3. The National Hydrometric Program is well managed according to established international operational criteria. The program is generally perceived to be well managed, as evidenced by the effectiveness of the NHP governance structure and bilateral agreements, the technical expertise and responsiveness of staff, and the effective introduction and maintenance of its ISO certification. However, as noted in section 3.2, the HSWS is experiencing some challenges related to program management (e.g., human resources planning, performance reporting) that will need to be addressed.

4. Recommendations

The recommendations of the evaluation of the Hydrological Service and Water Survey program are addressed to the Assistant Deputy Minister (ADM) Meteorological Service of Canada, as the ADM responsible for the two directorates accountable for program management and delivery, Weather and Environmental Monitoring, and Weather and Environmental Operations.

Recommendation 1: Develop a program logic model and performance measurement strategy to guide performance measurement and reporting.

Recommendation 2: Investigate and implement ways to improve the program’s understanding of user needs and demand trends in order to support a risk-based approach for the ongoing management of the network of gauging stations.

Recommendation 3: Review the current approach to human resource planning and staffing for professional and management positions, and develop an integrated program-wide plan to guide the attraction, retention and deployment of such staff.

5. Management Response

The responsible ADM agrees with all three recommendations and has developed a management response that appropriately addresses each of the recommendations. The full management response can be found in section 6 of the report.