TY  - CONF
AU  - Semcheddine, Asma
AU  - Lessig, Christian
AU  - Luise, Ilaria
AU  - Schultz, Martin
AU  - Langguth, Michael
AU  - Melidonis, Savvas
TI  - The Effect of BERT Training on Atmospheric Data Interpolation
M1  - FZJ-2025-03482
PY  - 2025
AB  - Atmospheric science has witnessed a breakthrough in recent years by harnessing deep learning models to understand and replicate  the complex relationships within and between different atmospheric variables. Atmorep [1], a foundational model of atmospheric dynamics, was developed as a task-agnostic model, trained on 40 years of hourly data in a BERT-style manner, with up to 90% of the data being masked, in order to provide a plethora of  downstream applications. To further assess the model's ability to learn a comprehensive abstract representation of atmospheric data, we tested several systematic token-masking strategies (geographical masking, temporal masking, a hybrid pattern combining both, and masking along model levels) and examined their effects on its data interpolation performance. Our preliminary results indicate that the coupled-fields transformer slightly outperforms the single-field transformer, reinforcing the correlation between different atmospheric fields. At a 75% compression ratio, AtmoRep achieves good reconstruction for the temperature field and all three wind components. Additionally, AtmoRep appears to benefit from the hybrid masking pattern, offering further insights into large-scale representation learning and enhancing our understanding of data-driven atmospheric modeling.
T2  - Dynamics Days Europe 2025
CY  - 23 Jun 2025 - 27 Jun 2025, Thessaloniki (Greece)
Y2  - 23 Jun 2025 - 27 Jun 2025
M2  - Thessaloniki, Greece
LB  - PUB:(DE-HGF)1
UR  - https://juser.fz-juelich.de/record/1045000
ER  -