Greenhouse Gas Emissions Forecasting: Learning from International Best Practices

While the case studies yield many interesting findings in relation to best practices in GHG emissions forecasting, it is important to determine to what extent the case studies follow the best practices criteria discussed in sections 4.1 and 4.2. It is also important to understand how forecasting methodology and governance relate to one another. Is one dependent on the other? Does strong methodology alone lead to strong forecasts? Is it possible for countries to have accurate forecasts without strong governance in climate policy? Is strong governance a key factor in determining accurate forecasts?

As both the U.S. and U.K. approaches are subject to extensive peer review and rigorous analysis and constant updating, it is likely from a methodological perspective that they produce accurate emissions forecasts.

From a governance perspective, it is preferable to have one central agency or group undertaking or coordinating emissions forecasting. It does not, however, necessarily detract to have several sources of forecasts, arising from competition among forecasters and also from the characterization of model uncertainty facilitated by multiple models. This is only preferable, however, if the oft-cited case of multiple forecasters who do not communicate with each other, do not compare results, and do not learn from each others’ errors, is avoided. But multiple forecast groups who talk, compare, and learn may produce better forecasts than just one group. Only in this circumstance, but still with the coordination of a central agency, could a multi-source approach to emissions forecasting be useful.

It is also important to bear in mind that different agencies within government obviously have different problems to solve and different policy and program objectives, so that different models may help get the appropriate answers they individually need.^[35] This does not mean, however, that this should be replicated for the complex, integrated challenges of large-scale GHG emissions forecasting, which is the Canadian government’s current approach. However, instead of individual departments submitting their estimates to Environment Canada, a preferable solution within this approach would be to require the different departments that produce the various forecasts to meet regularly to compare results and attempt to arrive at consensus on common questions of interest, or at least learn from others’ attempts. Even if no consensus is possible, the variety of results might provide an indication of uncertainty attributable to a lack of knowledge of the primary model.

In the area of governance, both the U.S. and U.K. received praise from the UNFCCC for the consistency in their approaches to forecasting. Aside from the U.S. and U.K., it is important to note the recent commitments by the other governments studied for the purposes of this report but not included as best-practice countries; the Norwegian and Australian governments have made commitments to independent peer review, centralized forecasts, and developed ambitious climate policy (including aggressive reduction targets) in general. This is especially of interest to Canada as these governments face similar economic and jurisdictional challenges.

With the exception of the U.K., the other case studies (U.S., along with Norway and Australia) reveal the lack of independent reviews by government auditors on the accuracy of emissions forecasts. In the case of the U.K., audits by an independent federal authority and an academic research institute can be said to improve understanding and credibility of its forecasts.

An important issue is the criticism levelled at all governments by the UNFCCC (and in the case of the U.K., its own independent auditors) on the significant forecasted emissions reductions attributed to voluntary measures. There is extensive literature on the limited success of attributing a level of specific emissions reductions as a result of voluntary measures. For example, studies highlight the paucity of credible evidence on the performance of voluntary programs compared to a realistic baseline.^[36] In its 2007 KPIA Response, the NRTEE noted that with "few exceptions, little evidence exists through which one can evaluate the incremental effect of information-provision programs for emissions control or energy conservation."^[37] This reliance on voluntary programs can therefore raise questions about the likelihood that the emissions forecasts from these measures will be accurate. For example, past emissions forecasts in Canada have relied on voluntary measures and programs to deliver significant reductions and in all cases have overestimated the forecast emissions reductions. Attributing significant emissions reductions to voluntary measures will by their very nature continue to result in inaccurate emissions forecasts. In this way, the governance and methodological issues raised in our analysis point to the need for policy makers to consider alternative climate policy measures to achieve desired GHG emissions reductions.

How strong is the link between good forecasting methodology and good forecasting governance in producing accurate emissions forecasts? From a governance perspective, all countries studied for the purposes of this report reveal that strong, central departments or agencies with independence should be responsible for emissions forecasting. In the case of the U.S., an independent statistical agency is responsible for producing its country’s GHG emissions forecasts.^[38] Where there is not an independent statistical agency responsible for forecasting (e.g. the U.K.), there is strong political commitment and centralized governance structures in place. Therefore, a significant finding (as noted in section 4.4.2) is that unless the agency in charge of forecasting GHG emissions has the authority or independence to make its own professional conclusions about the effectiveness of individual programs, there is a potential for the forecasts to be driven by more narrow program-driven considerations rather than a more independent and integrated analysis aimed at overall GHG emissions forecasting.

5.1 Lessons for Canada

While the above discussion, and key findings summarized in the conclusion below, provide important insight for Canada on potential areas to explore in its approach to emissions forecasting, it is important to determine key lessons from the case studies that can be applied to the Canadian context.

Independence of Agency/Department Responsible for Forecasting

In the U.S., as in other countries, individual departments or agencies routinely prepare analyses of specific GHG reduction programs – typically of programs they themselves administer. For example, the U.S. EPA routinely projects emission reductions resulting from its voluntary programs. While these agency analyses are reviewed by the EIA, they are not necessarily adopted by the EIA. This is an important point. Unless the agency in charge of forecasting GHG emissions has the authority/independence to make its own professional judgments about the effectiveness of individual programs, there is a potential for the forecasts to be driven by programmatic rather than analytic considerations. A perceived strength of the EIA is that it can and does have the authority and independence to disagree with other agencies on issues of this sort.

At the federal level in Canada, the closest existing body in terms of governance to the EIA would be Statistics Canada. While it does not currently undertake emissions forecasting, it collects emissions data for Environment Canada’s GHG emissions inventory. Through legislation, Statistics Canada could be given the responsibility and sole authority to produce Canada’s emissions forecasts. The independence of Statistics Canada would ensure that emissions forecasting would be free from interference and that it would have the authority to question and if necessary reject emissions reductions estimates from government departments. Alternatively, the federal government could also follow the example of Australia and create a new federal department or agency responsible for all climate change policy, including forecasting.

Clearly Articulated Legislated Roles, Responsibilities, and Milestones for Climate Policy

The U.K. Climate Change Bill, which is currently being subjected to a full public consultation alongside pre-legislative scrutiny in Parliament, sets out a fundamentally new and more structured approach to the setting of emissions reduction targets and the monitoring of performance against them. It will translate the goal the government announced in the 2003 White Paper into legislation--namely, a 60 per cent reduction in carbon dioxide by 2050 against a 1990 baseline. It will create a system of five-year carbon budgets to place the U.K. on a trajectory to meet its long-term goal and require the Secretary of State to set and meet carbon budgets for up to

15 years in advance. A newly created, external Committee on Climate Change will be responsible for advising the government on the level of carbon budgets to be set, and for monitoring emissions through annual reports to Parliament that will include a more comprehensive assessment of performance after the end of each five-year budgetary period.

Canada should consider a truly integrated long-term climate policy with similar characteristics. Given the comprehensive nature of the climate challenge, which crosses a range of government departments (not just environment), a central policy development and coordination body that links policy and program choices with integrated emissions forecasting would likely result in more confidence in the efficacy of proposed emissions reduction measures. Legislating its role and mandate to incorporate independent forecasting and evaluations would further strengthen its effectiveness and confidence in the policy approaches being pursued. Providing an external review or audit of emission forecasts would complete this new approach.

The Role of Provinces and Territories

An original intention of this report was to highlight best practices in jurisdictions other than national-level governments, particularly some Canadian provinces and U.S. states. Given the unique jurisdictional issues of climate policy in Canada, and the fact that 16 megatonnes of annual emissions reductions in Turning the Corner are attributed to provincial and territorial initiatives, it is important to determine if provinces are conducting accurate forecasts of their climate policies and measures.

With its past two reports on long-term issues related to climate change and energy,^[39] the NRTEE has consistently called for a coordinated, national approach to climate change issues in Canada. If the federal government is to produce accurate national emissions forecasts and ensure adopted policy measures achieve forecast reductions, there needs to be better coordination and understanding of projected emissions reductions from provinces’ climate policies and measures. There also needs to be a better understanding among governments in Canada as to various approaches to forecasting and how consistent methods can be applied to ensure accurate forecasts of GHG emissions reductions. Provinces should be encouraged to develop and release detailed emissions forecasts to inform their own policy choices necessary to meet the reduction targets they have set for themselves.

Scenario-Based Emissions Policy Forecasting

The U.S. EIA’s reference case projections are based on current laws and regulations with additional forecasts incorporating future policies. This issue is particularly important for Canada, as one of the concerns noted in section 4.3.2 is that the Turning the Corner modelling analysis assumed that "provincial mitigation policies improve over time and become more consistent between provinces." As noted in the NRTEE’s 2008 KPIA Response, potential future emissions reductions from provincial actions are counted as realized emissions reductions.

Scenario-based emissions policy forecasting can help address this concern. Future Canadian analyses might consider starting with projections of GHG emissions based on existing laws and policies and then explicitly add scenarios to reflect assumptions about improvements in provincial policies. This same, scenario-based approach can be extended to policy-level analysis, presenting a forecast of economy-wide emissions with and without the policy in place, each in a scenario where all other policies are in place. This provides an estimate of the marginal or incremental effect of the policy including adjustments for free-ridership and policy interaction effects, and depending on the structure of the model, rebound effects. These estimates could be included along with the current policy-level analysis or as replacement for them. Such an approach would, at a minimum, increase transparency and facilitate evaluation of future forecasts.

35 For example, a specific program or measure (i.e., fuel efficiency vehicle programs) may require a different forecasting model than one designed for large-scale, integrated modelling.

36 Morgenstern, D. and W.A. Pizer (eds) (2007), Reality Check: The Nature and Performance of Voluntary Environmental Programs in the United States, Europe and Japan. Washington, D.C.: RFF Press.

37 NRTEE (2007), Response of the National Round Table on the Environment and the Economy to its Obligations under the Kyoto Protocol Implementation Act, p. 32.

38 Statistics Canada -- Canada’s independent, federal statistics agency -- collects emissions data for the purpose of reporting, not forecasting.

39 NRTEE (2006), Advice on a Long-term Strategy on Energy and Climate Change; NRTEE (2008), Getting to 2050: Canada’s Transmission to a Low-Emission Future.