Forecast accuracy and efficiency at the Bank of England – and how forecast errors can be leveraged to do better – Bank Underground

Date:

Share post:


Derrick Kanngiesser and Tim Willems

This post describes a systematic way for central banks to employ past forecasts (and associated errors) with the aim of learning more about the structure and functioning of the economy, ultimately to enable a better setting of monetary policy going forward. Results suggest that the Monetary Policy Committee’s (MPC’s) inflation forecast has tended to underestimate pass-through from wage growth to inflation, while also underestimating the longer-term disinflationary impact of higher unemployment. Regarding the effects of monetary policy, our findings suggest that transmission through inflation expectations has played a bigger role than attributed to it in the forecast.

A sequence of unprecedented global shocks has recently posed a major challenge to economic forecasters across the world. Resulting forecast errors, particularly on inflation, have put central bank forecasts in the spotlight (see Bernanke (2024)).

The Bank of England MPC’s forecasts are constructed by drawing on a range of models, as described in Burgess et al (2013), augmented by staff and committee judgement. This raises questions of whether and how underlying forecast processes have contributed to forecast errors. In this regard, this post (which is based on our accompanying Staff Working Paper) proposes a strategy to use past Bank forecast errors to learn more about the UK economy. Generally speaking, out-of-sample forecasts are a good way to test the underlying forecast-generating model, which is what our approach leverages.

The data

At the heart of our exercise lies a data set of the Bank of England MPC’s quarterly forecasts from 2011 Q4 until 2024 Q1 for CPI inflation, wage growth, and real GDP growth (all annual rates, calculated year on year) as well as for the unemployment rate. In addition, we also use the yield on three-year UK government bonds, to analyse the impact of financial conditions (as shaped by UK monetary policy). We start our sample in 2011 Q4, as that date marks the adoption of the Bank’s current forecasting platform (Burgess et al (2013)). 

Chart 1 plots the outturns for the year-on-year CPI inflation rate (in dark blue), while the light blue lines depict MPC forecasts made at various points in time. From late-2021 onward, one can observe repeated upside surprises as inflation continued to rise; the disinflation process – which started late-2022 – is, thus far, developing more in line with forecasts.

Chart 1: UK CPI inflation, outturns and modal forecasts

Forecast accuracy

A first test that we can consider is whether the MPC’s forecast have systematically under or overestimated key variables. That is, in statistical parlance, whether there is sign of any ‘bias’ in forecast errors.

Chart 2 shows the mean (blue line) and median (red line) forecast errors for inflation, wage growth, unemployment and GDP growth. We have defined the h quarter-ahead forecast error for variable x_{t} as the difference between the forecast made in period t-h, F_{t-h} (x_{t}), and the ‘outturn’ for period t, x_{t},: FE_{t-h} (x_{t})=F_{t-h} (x_{t})-x_{t}. The grey swathe depicts the 68% percentiles. All forecast errors have been rescaled by the respective pre-pandemic sample standard deviation to facilitate comparability across variables. Since mean errors are heavily influenced by outliers (like those driven by the Covid shock or the increase in energy prices following Russia’s invasion of Ukraine), we will mostly focus on medians.

The median forecast errors on inflation have been very close to zero across all horizons (mean errors – which are more sensitive to outliers – point towards an underprediction of inflation). That is to say that, despite recent forecast errors (stemming from having underestimated the pick-up in inflation following Russia’s invasion of Ukraine), there is no evidence of a systematic bias in the inflation forecast over a longer sample. At the same time, the medians in Chart 2 also show that the MPC’s forecast has tended to overpredict wage growth, unemployment and GDP growth.

Chart 2: Average forecast errors over 2011 Q4–2024 Q1

How forecasts can be leveraged to learn more about economic relationships

A key concept in the forecast evaluation literature is that of ‘forecast efficiency’. It implies that the forecast appropriately uses all information that was available to the forecaster at the time the forecast was made. A testable implication is that forecast errors should not be predictable using information available to the forecaster at the time the forecast was produced. Otherwise, the forecaster could have made a more accurate forecast by using that information.

An important observation in this regard was made by Blanchard and Leigh (2013), who noted that one potentially relevant piece of information available to the forecaster are forecasts of other variables. Based on this insight, they devised a strategy to see whether forecasters over or underestimate the strength of certain relationships within the economy. If a (correctly forecasted) movement in a certain driving variable (say, wage growth) is systematically associated with higher-than-forecasted inflation two years later, then the MPC’s forecast can be said to underestimate the impact of wage growth on inflation at the two-year horizon.

Our objective is to test whether the MPC’s forecast systematically over or underestimates the strength of the relationships between certain driving variables and inflation. We therefore regress forecast errors (FE_{t-h} (y_t) which is the forecast error on variable y at time t, based on the forecast made h quarters ago) on two-quarter ahead MPC forecasts of variable x (F_{t-h} (x_{t-h+2}), but results are robust to other horizons):

FE_{t-h} (y_t) = \alpha_h + \beta_h F_{t-h} (x_{t-h+2}) + \epsilon_{h,t}

Here, x represents, alternatively, the unemployment rate, wage growth, real GDP growth, or the three-year yield on UK government bonds. We estimate (1) using the method of ‘robust regression‘, which down-weights observations that are considered ‘atypical’ (in terms of the regression not producing a good fit to the data, for example during the Covid-period).

When looking at inflation outturns on the left-hand side of (1), a negative estimate for \beta_{h} would suggest that the forecast methodology underestimates the inflationary impact of x at horizon h (ie, the forecast embeds an implied pass-through coefficient, from variable x to inflation, that is too low). A positive estimate for \beta_{h} would suggest the opposite. While we will discuss results in terms of an increase in the driving variable x, our regressions are symmetric – meaning that our findings also apply to decreases in the driving variable (but with the opposite sign).

The black solid lines in Charts 3 and 4 show the estimates of \beta_{h} for each horizon h equal to 1, 2, 4, 8 or 12; the shaded areas represent 90% confidence bands.

Chart 3: Blanchard-Leigh results for inflation forecast errors

Chart 3A shows that the estimated coefficient is negative at the two and three-year horizon. This means that forecasted wage growth is followed by inflation outturns that are higher than forecasted at the two and three-year forecasting horizon (where the grey confidence bands are different from zero). This suggests that MPC forecasts have underestimated the link from wage growth to inflation at those medium-term horizons.

Along similar lines, Chart 3B suggests that, at the two and three-year horizon, greater unemployment tend to be followed by lower-than-forecasted inflation outturns. This suggests that increases in unemployment might do more to lower inflation than historically encapsulated within the MPC’s forecast, which is for example consistent with the actual Phillips curve being steeper than encapsulated in the forecasting process.

According to Chart 3C, forecasted increases in GDP growth give rise to inflationary surprises up to horizons of two years, followed by lower-than-forecasted inflation at the three-year horizon. This could be due to the prospect of strong growth leading to some demand-driven inflation in the short run (more than forecasted), which is compensated by lower-than-forecasted inflation in later years (eg, thanks to increased supply arriving on the market). Alternatively, it could also be the case that policy reacts (more than was anticipated in the forecast) to the growth acceleration, responding in a way that tends to lower inflation (eg, tightening the fiscal or monetary policy stance).

Chart 4 reports equivalent findings for forecasted changes in three-year government bond yields. Since the latter are shaped by monetary policy, this exercise gives us an idea as to whether MPC forecasts have worked with an appropriate view of the monetary transmission mechanism. In this regard, it should be noted that the MPC forecast is not based on the MPC’s own forecast regarding the future path of interest rates, but is instead conditional on market-based interest rate forecasts. Results suggest that, relative to what has been encapsulated by the forecast, higher interest rates: A) have a stronger disinflationary impact over all horizons; B) push up unemployment by less at the three-year horizon; C) do less to slow real GDP growth at the two-quarter horizon; and D) do more to slow wage growth at the two and three-year horizon. Jointly, these observations are consistent with transmission through medium-term inflation expectations having played a bigger role than attributed to it in the forecast (as the inflation expectations channel can reduce inflation and wage growth without having to rely on a significant increase in unemployment; see Burr and Willems (2024)).

Chart 4: Blanchard-Leigh results for the monetary transmission mechanism

Conclusion

In this post, we have laid out a strategy through which central bank forecasts can be used to learn more about relationships between key variables of interest to the setting of monetary policy.

Our analysis suggests that there is scope for improvement by correcting certain relationships between variables within the MPC’s forecast. In particular, the pass-through from wage growth to inflation may be higher than assumed, while forecasts appear to have underestimated the longer-term disinflationary impact of higher unemployment. Finally, results suggest that monetary policy transmission via inflation expectations has played a bigger role than attributed to it in the forecasting process.

We hope these findings will be helpful in informing future forecast approaches, enabling improved setting of monetary policy.


Derrick Kanngiesser works in the Bank’s Monetary Policy Outlook Division and Tim Willems works in the Bank’s Structural Economics Division.

If you want to get in touch, please email us at bankunderground@bankofengland.co.uk or leave a comment below.

Comments will only appear once approved by a moderator, and are only published where a full name is supplied. Bank Underground is a blog for Bank of England staff to share views that challenge – or support – prevailing policy orthodoxies. The views expressed here are those of the authors, and are not necessarily those of the Bank of England, or its policy committees.

https://www.highcpmgate.com/f0c2i8ki?key=d7778888e3d5721fde608bfdb62fd997

LEAVE A REPLY

Please enter your comment!
Please enter your name here

Related articles

Big Bank Earnings Are About to Shake up Wall Street

What to know as BlackRock, JPMorgan, Wells Fargo, Citi and more all report earnings.

How to Make Money Writing Obituaries

When we think about income streams, the funeral industry might not be the first that comes to...

3 Reasons I Switched to a High-Yield Savings Account

There was a time when I, a financial writer, kept my savings in my checking account. I...