%0 Conference Paper
%A Semcheddine, Asma
%A Lessig, Christian
%A Luise, Ilaria
%A Schultz, Martin
%A Langguth, Michael
%A Melidonis, Savvas
%T The Effect of BERT Training on Atmospheric Data Interpolation
%M FZJ-2025-03482
%D 2025
%X Atmospheric science has witnessed a breakthrough in recent years by harnessing deep learning models to understand and replicate  the complex relationships within and between different atmospheric variables. Atmorep [1], a foundational model of atmospheric dynamics, was developed as a task-agnostic model, trained on 40 years of hourly data in a BERT-style manner, with up to 90% of the data being masked, in order to provide a plethora of  downstream applications. To further assess the model's ability to learn a comprehensive abstract representation of atmospheric data, we tested several systematic token-masking strategies (geographical masking, temporal masking, a hybrid pattern combining both, and masking along model levels) and examined their effects on its data interpolation performance. Our preliminary results indicate that the coupled-fields transformer slightly outperforms the single-field transformer, reinforcing the correlation between different atmospheric fields. At a 75% compression ratio, AtmoRep achieves good reconstruction for the temperature field and all three wind components. Additionally, AtmoRep appears to benefit from the hybrid masking pattern, offering further insights into large-scale representation learning and enhancing our understanding of data-driven atmospheric modeling.
%B Dynamics Days Europe 2025
%C 23 Jun 2025 - 27 Jun 2025, Thessaloniki (Greece)
Y2 23 Jun 2025 - 27 Jun 2025
M2 Thessaloniki, Greece
%F PUB:(DE-HGF)1
%9 Abstract
%U https://juser.fz-juelich.de/record/1045000