guest ::
| Home > Publications database > Dynamic load/propagate/store for data assimilation with particle filters on supercomputers > print |
| Home > Publications database > Dynamic load/propagate/store for data assimilation with particle filters on supercomputers > print |
| 001 | 1037349 | ||
| 005 | 20250203124526.0 | ||
| 024 | 7 | _ | |2 doi |a 10.1016/j.jocs.2024.102229 |
| 024 | 7 | _ | |a WOS:001185868400001 |2 WOS |
| 037 | _ | _ | |a FZJ-2025-00662 |
| 082 | _ | _ | |a 004 |
| 100 | 1 | _ | |0 P:(DE-HGF)0 |a Friedemann, Sebastian |b 0 |e Corresponding author |
| 245 | _ | _ | |a Dynamic load/propagate/store for data assimilation with particle filters on supercomputers |
| 260 | _ | _ | |a Amsterdam [u.a.] |b Elsevier |c 2024 |
| 336 | 7 | _ | |2 DRIVER |a article |
| 336 | 7 | _ | |2 DataCite |a Output Types/Journal article |
| 336 | 7 | _ | |0 PUB:(DE-HGF)16 |2 PUB:(DE-HGF) |a Journal Article |b journal |m journal |s 1737022774_7816 |
| 336 | 7 | _ | |2 BibTeX |a ARTICLE |
| 336 | 7 | _ | |2 ORCID |a JOURNAL_ARTICLE |
| 336 | 7 | _ | |0 0 |2 EndNote |a Journal Article |
| 520 | _ | _ | |a Several ensemble-based Data Assimilation (DA) methods rely on a propagate/update cycle, where a potentially compute intensive simulation code propagates multiple states for several consecutive time steps, that are then analyzed to update the states to be propagated for the next cycle. In this paper we focus on DA methods where the update can be computed by gathering only lightweight data obtained independently from each of the propagated states. This encompasses particle filters where one weight is computed from each state, but also methods like Approximate Bayesian Computation (ABC) or Markov Chain Monte Carlo (MCMC). Such methods can be very compute intensive and running efficiently at scale on supercomputers is challenging. This paper proposes a framework based on an elastic and fault-tolerant runner/server architecture minimizing data movements while enabling dynamic load balancing. Our approach relies on runners that load, propagate and store particles from an asynchronously managed distributed particle cache permitting particles to move from one runner to another in the background while particle propagation proceeds. The framework is validated with a bootstrap particle filter with the WRF simulation code. We handle up to 2555 particles on 20,442 compute cores. Compared to a file-based implementation, our solution spends up to 2.84 less resources (cores×seconds) per particle. |
| 536 | _ | _ | |0 G:(DE-HGF)POF4-5111 |a 5111 - Domain-Specific Simulation & Data Life Cycle Labs (SDLs) and Research Groups (POF4-511) |c POF4-511 |f POF IV |x 0 |
| 536 | _ | _ | |0 G:(EU-Grant)824158 |a EoCoE-II - Energy Oriented Center of Excellence : toward exascale for energy (824158) |c 824158 |f H2020-INFRAEDI-2018-1 |x 1 |
| 700 | 1 | _ | |0 P:(DE-HGF)0 |a Keller, Kai |b 1 |
| 700 | 1 | _ | |0 P:(DE-Juel1)164851 |a Lu, Yen-Sen |b 2 |u fzj |
| 700 | 1 | _ | |0 P:(DE-HGF)0 |a Raffin, Bruno |b 3 |
| 700 | 1 | _ | |0 P:(DE-HGF)0 |a Bautista-Gomez, Leonardo |b 4 |
| 773 | _ | _ | |0 PERI:(DE-600)2557360-3 |a 10.1016/j.jocs.2024.102229 |p 102229 |t Journal of computational science |v 76 |x 1877-7503 |y 2024 |
| 856 | 4 | _ | |u https://juser.fz-juelich.de/record/1037349/files/Dynamic%20load_propagate_store%20for%20data%20assimilation%20with%20particle%20filters%20on%20supercomputers.pdf |y Restricted |
| 909 | C | O | |o oai:juser.fz-juelich.de:1037349 |p openaire |p VDB |p ec_fundedresources |
| 910 | 1 | _ | |0 I:(DE-588b)5008462-8 |6 P:(DE-Juel1)164851 |a Forschungszentrum Jülich |b 2 |k FZJ |
| 913 | 1 | _ | |0 G:(DE-HGF)POF4-511 |1 G:(DE-HGF)POF4-510 |2 G:(DE-HGF)POF4-500 |3 G:(DE-HGF)POF4 |4 G:(DE-HGF)POF |9 G:(DE-HGF)POF4-5111 |a DE-HGF |b Key Technologies |l Engineering Digital Futures – Supercomputing, Data Management and Information Security for Knowledge and Action |v Enabling Computational- & Data-Intensive Science and Engineering |x 0 |
| 914 | 1 | _ | |y 2024 |
| 915 | _ | _ | |0 StatID:(DE-HGF)0100 |2 StatID |a JCR |b J COMPUT SCI-NETH : 2022 |d 2025-01-07 |
| 915 | _ | _ | |0 StatID:(DE-HGF)0200 |2 StatID |a DBCoverage |b SCOPUS |d 2025-01-07 |
| 915 | _ | _ | |0 StatID:(DE-HGF)0300 |2 StatID |a DBCoverage |b Medline |d 2025-01-07 |
| 915 | _ | _ | |0 StatID:(DE-HGF)0199 |2 StatID |a DBCoverage |b Clarivate Analytics Master Journal List |d 2025-01-07 |
| 915 | _ | _ | |0 StatID:(DE-HGF)1160 |2 StatID |a DBCoverage |b Current Contents - Engineering, Computing and Technology |d 2025-01-07 |
| 915 | _ | _ | |0 StatID:(DE-HGF)0160 |2 StatID |a DBCoverage |b Essential Science Indicators |d 2025-01-07 |
| 915 | _ | _ | |0 StatID:(DE-HGF)0113 |2 StatID |a WoS |b Science Citation Index Expanded |d 2025-01-07 |
| 915 | _ | _ | |0 StatID:(DE-HGF)0150 |2 StatID |a DBCoverage |b Web of Science Core Collection |d 2025-01-07 |
| 915 | _ | _ | |0 StatID:(DE-HGF)9900 |2 StatID |a IF < 5 |d 2025-01-07 |
| 920 | _ | _ | |l yes |
| 920 | 1 | _ | |0 I:(DE-Juel1)JSC-20090406 |k JSC |l Jülich Supercomputing Center |x 0 |
| 980 | _ | _ | |a journal |
| 980 | _ | _ | |a VDB |
| 980 | _ | _ | |a I:(DE-Juel1)JSC-20090406 |
| 980 | _ | _ | |a UNRESTRICTED |
| Library | Collection | CLSMajor | CLSMinor | Language | Author |
|---|