Reliable weather forecasts are critical for the planning and management of a variety of social and economic activities, such as water management. To make such forecasts, numerical weather prediction (NWP) models have been developed. However, NWP models are limited in their ability to represent certain physical processes and initial conditions, and thus include inaccuracies, which can be improved through calibration. Effective calibration should aim to provide forecasts that are unbiased, reliable in ensemble spread, skillful, and no worse than climatology forecasts. Regarding this last point, there is a general lack of literature which evaluates calibration models based on their ability to generate forecasts that are coherent with climatology. A forecast should be at least as valuable to a user as compared to simply looking at the climatology distribution of an area. Therefore, when raw forecasts lose their skill (i.e. raw NWP forecasts worsen with longer lead times), the calibrated ensemble forecasts should increasingly approach observed climatology. Challenges arise given that newly operationalized NWP models usually don’t produce hindcasts, and instead often only archive a short period of experimental forecasts (typically of one year or less). In other words, the forecast data available to establish calibration models is often very limited.

To address these points and challenges, we developed a seasonally coherent calibration (SCC) model. The model was designed to account for seasonal variation in climatology and the relationship between observation and raw forecasts. When faced with limited raw forecasts, previous calibration methods have opted to neglect seasonality. This may be acceptable when the underlying skill of the raw forecasts is high. However, this breaks down at longer lead times. To account for this, SCC identifies some of the statistical characteristics of both raw forecasts and observations whilst making distinctions between different months. In addition, long records of observations are used to estimate parameters describing observed climatology. The remaining number of model parameters are reduced to a much smaller set. The idea behind reducing the number of parameters is to pool data together to estimate the remaining parameters, and to improve the computational efficiency of SCC.

Performance of SCC model at the site scale

To test the viability of the SCC model, we first used it to calibrate precipitation forecasts at one site in Queensland, Australia. This site was chosen given that precipitation in this area has strong seasonality. The figure below shows that the raw forecasts (blue and green lines – top plot) are clearly deviated from the observed climatology – see the solid and dashed red lines, which represent the 3- and 10-year average, respectively. In contrast, the calibrated forecasts (blue and green lines – bottom plot) are much more consistent with climatology in relative terms.

The forecasts were also evaluated based on their ability to achieve accurate frequency of wet days, daily distribution of precipitation, reasonable reliability (i.e. spread) of the forecast ensemble, as well as forecast skill. Improvements from SCC were noticeable for all evaluation criteria mentioned above. For an explanation of the full results of the Queensland case study as well as the detailed mathematical rationale behind SCC, please see Wang et. al., (2019). The positive results from this case study steeped our curiosity to apply the SCC model across a larger domain at a fine spatial resolution.

Application across the continent scale

Given the promising results we found for a singular site in Queensland, our next aim was to test the feasibility of using SCC for more general applications. Thus, we used SCC to produce ensemble forecasts for a larger domain (the whole of Australia). Despite the practical need for post-processing of NWP data at the continental scale, previous literature has tackled this by either calibrating at a coarse spatial resolution or has utilized a simple calibration method. Here we applied SCC, which can be considered a sophisticated calibration model, at a high spatial resolution (5km) on the continental scale. By doing this, our intent was to further test the robustness of SCC, as well as to check whether the computational demand of SCC would make it feasible for further wide-scale applications. The plot below shows average rainfall for raw (left plot) and SCC-calibrated (middle plot) day 1 ahead forecasts, compared with observation (right plot), averaged over the period of April 2016- March 2018. The SCC-calibrated forecasts of rainfall show a much-improved pattern, consistent with observation (although some of the strange patterns in sparsely populated areas are due to interpolation artifacts in the gridded dataset of observations).

One important reason for why SCC can be applied so easily at the continental scale, is that SCC reduces parameters, which helps to alleviate some of the computational burden. Our application across Australia shows that this reduction is valid and feasible for a wide range of climatic zones. The SCC model could be easily adapted to post-process other NWP variables, including temperature, wind, vapor pressure, solar radiation, and reference evapotranspiration. One obvious potential application of SCC is to post-process precipitation forecasts for the Australian continent to drive operational water balance forecasting models (e.g. see AWRA project: bom.gov.au/water/awra/). This could help reduce the uncertainties that plague water management decisions. SCC has the potential to benefit a range of forecast users by providing well-calibrated ensemble forecasts.

Find out more:

Wang Q. J., T. Zhao, Q. Yang and D. Robertson (2019), A Seasonally Coherent Calibration (SCC) Model for Postprocessing Numerical Weather Predictions. Mon. Wea. Rev., 147, 3633-3647, https://doi.org/10.1175/MWR-D-19-0108.1.

Stay tuned for:

Yang Q., Q. J. Wang, K. Hakala (in review), Achieving effective calibration of precipitation forecasts over a continental scale.

Originally published by HEPEX, 24 June 2020.