guest ::
| Home > Publications database > The Effect of BERT Training on Atmospheric Data Interpolation |
| Home > Publications database > The Effect of BERT Training on Atmospheric Data Interpolation |
| Poster (After Call) | FZJ-2025-03483 |
; ; ; ; ;
2025
This record in other databases:
Please use a persistent id in citations: doi:10.34734/FZJ-2025-03483
Abstract: Atmospheric science has witnessed a breakthrough in recent years by harnessing deep learning models to understand and replicate the complex relationships within and between different atmospheric variables. Atmorep [1], a foundational model of atmospheric dynamics, was developed as a task-agnostic model, trained on 40 years of hourly data in a BERT-style manner, with up to 90% of the data being masked, in order to provide a plethora of downstream applications. To further assess the model's ability to learn a comprehensive abstract representation of atmospheric data, we tested several systematic token-masking strategies (geographical masking, temporal masking, a hybrid pattern combining both, and masking along model levels) and examined their effects on its data interpolation performance. Our preliminary results indicate that the coupled-fields transformer slightly outperforms the single-field transformer, reinforcing the correlation between different atmospheric fields. At a 75% compression ratio, AtmoRep achieves good reconstruction for the temperature field and all three wind components. Additionally, AtmoRep appears to benefit from the hybrid masking pattern, offering further insights into large-scale representation learning and enhancing our understanding of data-driven atmospheric modeling.
|
The record appears in these collections: |
PDF