Abstract

FZJ-2026-05609

http://join2-wiki.gsi.de/foswiki/pub/Main/Artwork/join2_logo100x88.png Leveraging scale dependent accuracy measures for a fair comparison between AI and NWP weather prediction models

Seegebrecht, B. (Corresponding author) ; Wahl, S. ; Hollborn, S. ; Craig, G. ; Pavel, E.FZJ* ; Almikaeel, W.FZJ* ; Buschow, S.FZJ* ; Schultz, M. G.FZJ* ; Lessig, C. ; Luise, I. ; Al-Iahham, A. ; Gall, J.

2026

This record in other databases:  

Please use a persistent id in citations: doi:10.5194/ems2026-469

Abstract: Data-driven weather prediction models based on artificial intelligence (AI) have rapidly advanced in recent years and are frequently reported to outperform traditional physics-based numerical weather prediction (NWP) models for selected verification scores. However, optimization with respect to a specific loss function can adversely affect other metrics, potentially leading to unrealistic forecast characteristics, such as overly smooth spatial structures when mean-squared or mean-absolute error–based loss functions are used.In Bonavita & Geer, 2026 an orthogonal decomposition of the Mean Squared Error (MSE) into Information Error and Noise Error is introduced to unravel different strategies for minimizing this commonly used accuracy metric. These insights allow for a more meaningful interpretation of the MSE as accuracy measure.Additionally, a scale dependent analysis of model performance can help to reveal systematic differences between AI and NWP models specifically on smaller scales such as the effective resolution.The combination of both approaches – the decomposition of the scale dependent MSE into Information and Noise Error – has been derived and applied to different AI and NWP models.First results show the potential of this method to disentangle different effects allowing for a fairer, more comprehensive comparison between AI and NWP weather prediction models.The analysis is partly based on forecasts from the Weather Prediction Model Intercomparison Project (WP MIP), which provides a collection of NWP and AI-model forecasts from multiple national weather services and research institutions.The work is conducted within the RAINA project, which aims to develop a foundation model for the atmosphere with a particular focus on reliable, high-resolution forecasts of extreme wind and precipitation events.

---

Contributing Institute(s):

1. Jülich Supercomputing Center (JSC)

Research Program(s):

1. 5111 - Domain-Specific Simulation & Data Life Cycle Labs (SDLs) and Research Groups (POF4-511) (POF4-511)
2. BMFTR 16IS24075A - Ein statistisch robustes KI-Grundlagenmodell der Atmosphäre für bessere Kurzfristvorhersagen von Extremereignissen (RAINA) (16IS24075A) (16IS24075A)
3. Earth System Data Exploration (ESDE) (ESDE)

Appears in the scientific report 2026

---