guest ::
| Home > Publications database > Storing, Finding and Accessing Earth System Modeling Data: Insights and Challenges from WarmWorld-Easier |
| Home > Publications database > Storing, Finding and Accessing Earth System Modeling Data: Insights and Challenges from WarmWorld-Easier |
| Poster (After Call) | FZJ-2025-04377 |
; ; ; ;
2025
Abstract: Improvements in computational speed lead to better resolutions in Earth System Models (ESM) allowing them to resolve scales of a few kilometers. The volume of the resulting data greatly increases with the improvements in resolution and introduces a challenge to processing and storing these results. While modern HPC systems provide petabyte-scale capacity for file storage, analyzing such data on local user systems can become a prohibitive bottleneck. Beyond the sheer demands on processing high-volume ESM data, there is also increasing demand to make them FAIR and in particular findable. The goal of the “Easier” module of the Warmworld project aims to simplify the access to ESM data from different HPC centers, in particular the German Climate Computing Center (DKRZ) and the Jülich Supercomputing Centre (JSC). One aspect is the creation of a joint catalog following the SpatioTemporal Asset Catalogs (STAC) specification. This enhances the findability for data available at both centers. The accessibility is provided by links in the catalog to access the data, either directly on disk, as a download or, as a mid term goal, through streaming of data on demand with zarr over http. The explicit implementation of the required REST-APIs depends on the infrastructure, hardware and software, of the data centers as well as the organization of the stored data. As the first data backend at JSC, we set up a Fields DataBase (FDB), developed by European Centre for Medium-Range Weather Forecasts (ECMWF), to store the ESM results as multi-dimensional data cubes. For data retrieval, we provide a download service. Data are identified within the FDB by their metadata according to the Meteorological Archival and Retrieval System (MARS), developed by ECMWF. The automated access to HPC system is achived with the UNiform Interface to COmputing REsources (UNICORE). The download service integrates HelmholtzID, the Helmholtz authentication and authorization infrastructure. This will allow a large number of institutions to access these services with the possibility to control access and resource use. ƒOur poster will illustrate the combination of these various tools and services for creating the envisoned services for the infrastructure at JSC as well as our federated STAC catalog. We wish to close our presentation with an overview on further challenges related to ESM data addressing aspects of data management and data access.
|
The record appears in these collections: |
PDF