Abstract

FZJ-2025-03482

http://join2-wiki.gsi.de/foswiki/pub/Main/Artwork/join2_logo100x88.png The Effect of BERT Training on Atmospheric Data Interpolation

Semcheddine, A. (Corresponding author)FZJ* ; Lessig, C. ; Luise, I. ; Schultz, M.FZJ* ; Langguth, M.FZJ* ; Melidonis, S.FZJ*

2025

Abstract: Atmospheric science has witnessed a breakthrough in recent years by harnessing deep learning models to understand and replicate the complex relationships within and between different atmospheric variables. Atmorep [1], a foundational model of atmospheric dynamics, was developed as a task-agnostic model, trained on 40 years of hourly data in a BERT-style manner, with up to 90% of the data being masked, in order to provide a plethora of downstream applications. To further assess the model's ability to learn a comprehensive abstract representation of atmospheric data, we tested several systematic token-masking strategies (geographical masking, temporal masking, a hybrid pattern combining both, and masking along model levels) and examined their effects on its data interpolation performance. Our preliminary results indicate that the coupled-fields transformer slightly outperforms the single-field transformer, reinforcing the correlation between different atmospheric fields. At a 75% compression ratio, AtmoRep achieves good reconstruction for the temperature field and all three wind components. Additionally, AtmoRep appears to benefit from the hybrid masking pattern, offering further insights into large-scale representation learning and enhancing our understanding of data-driven atmospheric modeling.

---

Contributing Institute(s):

1. Jülich Supercomputing Center (JSC)

Research Program(s):

1. 5111 - Domain-Specific Simulation & Data Life Cycle Labs (SDLs) and Research Groups (POF4-511) (POF4-511)
2. Earth System Data Exploration (ESDE) (ESDE)
3. BMFTR 01LK2316A - Warmworld Smarter (IconRep) (-01LK2316A) (-01LK2316A)

Appears in the scientific report 2025

---